Boston Financial Data Services Business Continuity Executive Summary. November 2009

Transcription

1 Boston Financial Data Services Business Continuity Executive Summary

2 Boston Financial continues to maintain an active business continuity program that effectively supports the ability to survive a disruption to normal business operations by maintaining a continuous operations environment. This environment is supported locally and outside of the region, utilizing backup sites within DST s Winchester Data Center and Poindexter facilities. All operationally critical functions have been identified and are redundant in our disaster recovery location. Boston Financial has several operating facilities that can be used in a crisis. The philosophy is to provide an alternate workplace for our associates and clients during an event, without impacting the existing business units in the affected facility. Our recovery strategy includes a recovery center, located in the East, which has the capability to seat up to 375 associates using a combination of dedicated recovery seats and previously identified non-critical workstations. These workstation positions are equipped with AWD workstations and Call-Master/ACD phones as necessary. The recovery strategy also includes reciprocal agreements and regional recovery alternatives with our facilities in Kansas City, Missouri; Lawrence, Kansas; and Rockland, Maine. All facilities are equipped with backup generators capable of recovering 100 percent of the critical functions. The data center is equipped with a state-of-the-art UPS system. In the Midwest, alternate site capabilities have been identified using reciprocal strategies developed between the Boston Financial Business Continuity Group and DST Systems, relying upon the vast infrastructure and facility capabilities at DST to provide uninterrupted services to our clients. In addition, regional recovery strategies are in place with East Coast counterparts. Objectives The Boston Financial Contingency Plan has been developed to meet the following objectives: 1) Provide for the immediate, accurate and measured response to emergency situations; 2) Minimize the impact upon the safety and well being of personnel; and 3) Provide our clients with alternate site processing with a minimum of inconvenience. Contingency Planning Activities Boston Financial has a two-tiered approach to business recovery-corporate recovery and business line recovery. Our Business Recovery team is responsible for the recovery of corporate infrastructure. The business lines are responsible for executing recovery plans for their departments. These tools include, but are not limited to, various types of passive exercises from employee training and development, building desktop supply kits, round-table discussions, call trees and connectivity tests. Active exercises include but are not limited to, declaration, emergency response, hot site recovery workstation testing, and the relocation of business lines to alternate facilities. Infrastructure In the East, Boston Financial maintains a campus network infrastructure using Synchronous Optical Network technology (SONET) to provide alternate routing capabilities for allocations. The infrastructure is a fully redundant, high-speed fiber optic network. Voice communication requirements have been assessed in terms of recovery requirements for toll free numbers, direct inward dial, automatic call distribution, fax, and voice mail. Boston Financial has invested in AT&T s Route-IT services. This program allows Boston Financial s telecommunication personnel to reroute Boston Financial owned toll-free services throughout our campuses. At our request, our vendors have provided multiple conduits into all our facilities, as well as building a new route from the Boston to Quincy, MA. area. Our clients are part of the Boston Financial network recovery service contract, which provides them with a backup circuit that can be rerouted to Boston Financial s data center recovery site. 1

3 In our Midwest offices, identified reciprocal site strategies between the Boston Financial Business Continuity Group and DST are tested annually to maintain and provide uninterrupted services to our clients. Our Midwest strategy for continuous operations includes using the DST Full Service site and DST infrastructure to recover our operations. Testing, Procedures and User Manuals Boston Financial has developed comprehensive and evolving business recovery plans (BRPs) to address disruptions in service. The plans encompass Boston Financial s various lines of businesses and the technical platforms that support them. BRPs are updated quarterly using the Living Disaster Recovery Planning System (LDRPS) on-line planning tool. BRPs are exercised throughout the year based upon a schedule and in a manner agreed upon between the Corporate Business Continuity Planning (CBCP) department and the business unit. The CBCP department schedules, attends, and monitors exercises for each business unit. Results are documented between the CBCP department and the business unit and distributed to the client service officer of the applicable group. Our active exercises include, but are not limited to, declaration, emergency response, vendor participation, and relocation of business lines environment to an alternate facility. Each business unit is responsible for conducting an annual relocation exercise. For passive testing, each business unit is responsible for updating their offsite materials twice a year. Business units are required to execute notification (call tree) exercises at the minimum of twice a year, which includes participation in company-wide automated notification exercises using NotiFind. Boston Financial continues to improve and enhance the plans by partnering with the world s leading service and support vendors. The clients and subsidiaries of Boston Financial are part of the BRP s in place at Boston Financial. We revise, update, and test the plans for capacity, redundancy, and diversity demonstrating the ability to support our critical functions internally and with key affiliates. These critical functions are associated with the support and execution of mission-critical transaction processing by Boston Financial on DST s TA2000 and TRAC systems, as well as the AS/400 and the AWD Image environment. Boston Financial also participates in DST s semi-annual disaster recovery exercises where connectivity to DST s alternate hot sites is tested and operational functions are processed and emulated in a disaster recovery test program. Disaster recovery activities include emergency response, failure assessment, declaration (notification to third-party vendor of intent to occupy alternate site), alternate site implementation, and restoration of the primary site. Personnel Boston Financial s Continuity Plan relies upon its key operational and technical personnel performing predefined roles and responsibilities. All key positions related to business continuity have been identified and incorporated into the plan by the individual business unit. Notification call trees have been developed for these positions and are updated and tested as changes warrant, but at least quarterly. An associate information line has been established for corporate communications. The Event Management team s communications protocol is tested at least biannually. Training and Support Each business unit is required to designate a Business Continuity Planner (BC Planner). The CBCP department maintains a program to train and support associates who have been designated as BC Planners in various roles and responsibilities. In addition, all Boston Financial associates are required to participate in an annual online training program designed to raise the awareness level of business continuity planning and objectives throughout the company. 2

4 Disaster Recovery - Data In 2003, DST acquired a property in St. Louis, MO. The primary data center in Kansas City and the St. Louis recovery center are at an optimal geographic distance (250 miles) to avoid a shared natural disaster yet still close enough to supplement staff with associates from Kansas City. Both centers offer built-in security, redundancy, and recovery protocols to ensure a four-hour recovery across all avenues of system access, including interfaces to the Internet, NSCC, TA2000 Desktop, TA2000/Voice, and AWD. Additionally, we have installed data mirroring technology that continually replicates customer data so transactions from the current day are not lost as in the former pointin-time backup scenario. Both data centers have the capability to run indefinitely with backup generators and onsite storage of fuel and water and recovery exercises are performed at least quarterly. The backup data center is tested at least two times per year and fully recovers the Kansas City data center in four hours or less. Operations at the recovery site can continue indefinitely. Network connectivity for replicating data between the Production Data Center and the Recovery Data Center is engineered for high availability, including dual feeds into each building, to minimize the risk of network outages. The Recovery Point Objective (RPO) for TA2000 is 0 to 30 minutes. DST estimates that it will take no more than four hours to IPL our OS/390 LPARS and then restart our platforms on the processors in the Recovery Data Center. All processes should restart from the point that the failure occurred. The only data that could potentially be lost are transactions that are being entered at the actual time of the disaster. These transactions would have to be reentered. Contingency Planning Activities Disaster recovery activities include emergency response, failure assessment, declaration, recovery site implementation, and restoration of the primary site. Boston Financial s Internal Audit team reviews our plan periodically. Emergency Response In response to an emergency of any kind that could potentially affect the day-to-day operations of DST and/or its clients, predefined steps are in place and regularly tested by the Winchester Incident Management Team (IMT). The IMT will determine if the situation warrants declaration of a disaster. Disaster Recovery Exercises TA2000 recovery exercises will be performed a minimum of four times a year: two internal DST exercises and two external client exercises. All client platforms are recovered as part of the exercises. TA2000 recovery exercises last 24 hours: internal exercises run from 6:00 AM Friday through 6:00 AM Saturday. External exercises run from noon on Saturday through noon on Sunday. Client Participation In client participation exercises, remote clients who choose to participate via the network will be connected to the recovery data center. Clients who provide their own network to the Winchester Data Center will need to provide their own network to the Recovery Center. Clients can also choose to participate by requesting their DST client representatives to execute their testing or by traveling to the DST local recovery site in Kansas City. Automated Work Distributor Data Recovery The AWD Data Center provides disaster recovery through the use of a second AS/400 located at the Winchester Data Center. We use software from Lakeview Technologies to maintain a hot backup from the AS/400 located at our AWD Data Center to the AS/400 at the Winchester Data Center and EMC s Storage Area Network (SAN) solution to provide real-time backup of images. In the event of a disaster, the backup AS/400 located at the Winchester Data Center becomes the primary system. Workstations may be redirected to the backup AS/400 through desktop switcher code or through a centralized DNS entry change, depending on the client s desktop configuration preference. The switch process in our testing thus far has been accomplished in less than 10 minutes once the decision to switch has been made. Image replication on EMC may be switched to the redundant site independently of AS/400 systems. AWD peripheral servers are backed-up nightly. AWD recovery exercises are conducted twice per year, but participation is voluntary. 3

5 Emergency Levels / Recovery Scenarios Emergency Level Issue Recovery Scenarios Probability (H,M,L) Recovery Time Objective Level I Level II Level II a Level III Severe Weather Difficult to access, vacate or utilize building. Snow, flooding, heat-wave/brown out, etc. Pandemic Situation Excessive absenteeism Building Loss of Use-1 Day A single building becomes inoperable due to weather, and/or loss of power, water supply, air quality, telecommunications systems, etc. No Loss of physical building, equipment, records, etc. Loss of Use (2 or more buildings):1 Day Same as above Building Loss of Use-2-5 Days A single building becomes inoperable due to weather, serious power loss, air quality or bio-hazard health risk, structural systems failure, telecommunications loss, etc. Moderate loss of physical building, equipment, records, etc. Loss of Use (2 or more buildings): 2-5 Days Same as above. Communication Protocol Associate, equipment, facilities & client workload assessment. Decision & implementation Management team members. After Business Hours Notification Sequence: Security, Facilities, Event Management team. Communication Protocol Decision & implementation of short-term relocations. Management team members. After Business Hours Notification Sequence: Security, Facilities, Event Management team. Communication Protocol Short-term associate relocations required. Decision & Implementation Management team members. Notify affiliates and clients. After Business Hours Notification Sequence: Security, Facilities, Event Management team. High Medium High Medium 2 hours 2 hours 2 hours Level IV Level IV a Building Loss of Use-90 Days A single building becomes inoperable due to extreme/severe issue resulting in loss of use for an extended period of time. Loss of Use (2 or more buildings): 90 Days Same as above Communication Protocol Implementation of associate relocations for an extended period. Management team Members. notify affiliates and clients. After Business Hours Notification Sequence: Security, Facilities, Event Management team. Medium Low 4 hours Level V Building Loss of Use-90+ Days Extended loss of use of one building for an extended period due to geo-political or local/community wide issue. Loss of Use (2 or more buildings): 90+ Days Same as above Communication Protocol. Management team members. Notify affiliates and clients. After business hours Security, Facilities, Event Management team. Low 4-8 Hours

6 Business Unit Recovery Time Objective Mutual Fund Divisions Recon / Control Mail Services / Front End Operations TA2000 E Commerce 0 to 4 hours TA2000 Voice 0 to 4 hours Client Network Connectivity 0 to 4 hours Facilities Business Continuity Business & Technology Services DST Retirement Solutions Investor Services Compliance 2 to 4 hours Settlement Administration 2 to 4 hours Internal Audit 24 hours Human Resources 24 hours Learning & Organization Development 24 hours Finance Division 24 hours Marketing / Legal 24 hours The Recovery Time Objective for critical functions within each business unit is to be operational within 24 hours of a declaration. This window may decrease depending upon the time of disaster, availability of system connectivity, and impact to other corporate locations. Boston Financial s strategy for continuous operations includes the utilization of our multiple sites to recover each other. This is inclusive of telecommunications as well as data processing. Associates who are affected by the disaster will be relocated to another Boston Financial location to execute the critical processes associated with their business unit. Pandemic Flu Preparedness: Client Overview For informational purposes only: It does not create, alter, or amend any rights or obligations between any recipient of this document and Boston Financial. The potential for an influenza pandemic generates a great deal of concern in all areas of society. Assessing its impact is difficult since we can't predict with certainty when such a pandemic might start, its degree of virulence, or the public health and civil authorities ability to respond to it. A pandemic flu would most likely involve higher rates of infection and death than existing flu viruses, affecting every kind of activity. The Boston Financial Corporate Business Continuity Group, along with Human Resources, has been collaborating with DST Systems and State Street Corporation to identify the issues that a pandemic may raise and to develop responses to potential issues. All business units have updated their Business Recovery Plans to include pandemic planning information. The framework for Boston Financial's plan incorporates information and recommendations from the World Health Organization (WHO) and the Centers for Disease Control and Prevention (CDC), and other sources, including the federal Web site: Our framework outlines the possible extent and effects of a pandemic, the current state of vaccine development, availability of anti-virals, and possible actions to mitigate the effects. 5

7 Such actions may include: Cross-training of associates Allowing associates to work from home where possible and appropriate Dividing department staff to operate in multiple locations Moving work between existing staff in multiple locations Using social distancing techniques to minimize the spread of the virus In developing a pandemic response, Boston Financial followed some general assumptions: 1. The pandemic will impact everyone: associates, their families (i.e., schools, day care, etc.), clients, suppliers, utilities, hospitals, emergency services, federal, state and local government, transportation, financial services, retail, etc. 2. The pandemic will be global within 30 to 90 days from being designated by WHO. 3. The pandemic will occur in up to three waves, each wave lasting 8-12 weeks, with a total duration of up to 18 months. 4. Absenteeism from work during a pandemic wave will be percent and could crest at 50 percent. As part of Boston Financial's business continuity planning, the Corporate Business Continuity Group implemented requirements for a business impact analysis that includes pandemic planning for each business unit. The analysis focuses on identifying critical functions and documenting the depth of critical staffing within departments. Pandemic plans are updated as required, but no less than annually; they were most recently updated in the latter half of In addition, our vendors are required to provide readiness statements annually. As of the date of this document, if a pandemic were to occur, Boston Financial believes it will be able to continue operations. We base this belief on the general assumptions outlined previously, information currently available, and reviews of our continuity plans and annual exercises; however, there are major variables that remain unknown and may not become known until a pandemic emerges. Any one of these variables could impair the ability of organizations, including Boston Financial, to operate effectively. These include, but are not limited to: The degree of virulence of the pandemic disease The availability and degree of distribution of vaccines and anti-virals The ability of the federal, state, and local authorities to maintain services, including catastrophic response How the incidence of illness is distributed across Boston Financial's organization The ability of utilities to maintain services The availability of networks, particularly the Internet Legislation relative to pandemics and the operation of companies during such periods Imposition of isolation requirements (quarantine) by state and local authorities with little or no notice Boston Financial will continue to monitor developments relative to a pandemic disease, modify plans accordingly, and periodically update and distribute this client overview as conditions warrant. Client inquiries relative to specific situations should be directed to the client's relationship managers Pandemic Planning Information Business Continuity Planning, Human Resources, and Facilities Management, as well as DST Systems and DST Output, have reviewed Boston Financial s approach to any potential pandemic outbreak. All business units were required to update their Business Recovery Plans based on the following criteria. 6

8 Business lines are open (internal & external) Mail processing is at 75 percent (inbound mail) Individual business groups are experiencing an absenteeism rate of 40 percent Support groups are also experiencing 40 percent absenteeism rate Absenteeism lasts 6-8 weeks Business units are also required to review and update their business recovery documents quarterly, including considerations for pandemic planning. As part of the review, each business unit is asked to perform the following exercises, including the above assumptions: 1. Tabletop exercise 2. Call tree exercise 3. Relocate to disaster recovery site The senior management team at Boston Financial reviews all business contingency information and testing results. 7