DEPARTMENT OF LABOR AND EMPLOYMENT DISASTER RECOVERY AND BUSINESS RESUMPTION CONTINGENCY PLANNING GUIDELINES

Transcription

1 COLORADO DEPARTMENT OF LABOR AND EMPLOYMENT 1515 Arapahoe Street Denver, Colorado SUPPORTING DOCUMENTATION Title: Disaster Recovery Plan Related Document: Disaster Recovery Plan Development Source: Author: Mel Madden Effective Date: 01/20/94 Number of Pages: Copyrighted: Yes No DEPARTMENT OF LABOR AND EMPLOYMENT DISASTER RECOVERY AND BUSINESS RESUMPTION CONTINGENCY PLANNING GUIDELINES

2 PART I OVERVIEW Although the term disaster usually implies widespread destruction, the term disaster recovery, as used in this document, refers to all activities which must be initiated (or re-initiated) in response to an unanticipated and prolonged disruption in normal business activities. Continuity of operations throughout the business activities of the Department is critical to meeting the service delivery goals of the Department Any number of occurrences may disrupt normal business activities for a period of time ranging from an hour to a month or more. It is essential that Business Program Managers analyze their business processes and develop contingency plans which will ensure that all negative aspects of any disruption are addressed immediately and effectively to minimize the potential impact on clients, employees, the Department, and the State of Colorado. This document discusses the various categories/levels of business disruptions and the pre-planning which must take place to ensure an orderly, phased resumption of business activities when a disruption occurs. Note that many of the planning considerations presented apply to manual business processes as well as business processes which are supported by automated systems. PART II POLICY STATEMENT It is the policy of the Colorado Department of Labor and Employment that Disaster Recovery and Business Resumption Contingency Plans will be prepared and reviewed or tested for all business activities that are considered critical to the continued operations of any business program in the event of a disaster or disruption of normal business activities. Plans will be prepared and administered according to the guidelines presented in this document. PART III DEFINITIONS DISASTER: An occurrence that impacts one or more business activities of the Department to the extent that the capability to perform normal operations is impaired. DISASTER RECOVERY: The resumption of business activities following a disaster.

3 DISASTER RECOVERY PLAN: A documented sequence of activities that will ensure an orderly, phased resumption of business activities following a disaster. PART IV PLANNING CONSIDERATIONS, GGCC FACILITIES FAILURE This part describes the activities that are the responsibility of the Department in the planning for and response to a disaster caused by a failure of facilities controlled by GGCC which is expected to last more than 3 days. A. Identification of Critical Business Activities which cannot be performed in some alternate manner (other than using mainframe days. automated support systems) for a period of 14 consecutive calendar 1. Identification of business activities is the responsibility of the Business Program Manager. 2. For each business activity so identified, the Business Program Manager must prepare a justification which discusses in detail why the activity cannot be performed in some alternate manner. Include a description of all alternatives considered and the rationale which led to the conclusion that they were not feasible. 3. The Information Systems Steering Committee will approve or disapprove the designation of a Critical Business Activity, based on the justification provided by the Business Program Manager. 4. The Information Systems Steering Committee will assign priorities for recovery to all designated Critical Business Activities. B. Identification of automated systems or sub-systems which support the Critical Business Activities. The Office of Information Systems will assist the Business Program Manager in identifying the specific systems or sub-systems which support the Critical Business Activities.

4 C. Identification of CDLE controlled data files, applications programs, and associated items required to support the systems or sub-systems. 1. The Office of Information Systems will identify the specific applications software programs, the data files, and the JCL and related files which must be available to recover the identified systems/sub-systems. 2. The Office of Information Systems is responsible for ensuring that current copies of all required files are maintained, at all times, at a backup site not on the GGCC premises. D. Identification of application system reports, logs, etc., required to support non-critical business activities. 1. The Business Program Manager will provide OIS with a full description of all items which must be made available to the work unit to use as the starting point for manual operations which will be put into place while the more critical systems continue to process at the hot site. 2. OIS must ensure that existing job streams are modified to generate (for off-site backup storage) the necessary products (i.e., a tape copy of a print file which contains a current master list) to be used to produce the required documents. E. Identification of files and programs controlled by other agencies which are required to support the systems/sub-systems. The Office of Information Systems will identify any applications programs or files belonging to other agencies which must be available for interface from the identified systems. EXAMPLE; The COFRS Vendor File, owned and controlled by another agency, must be available and active in order for the Special Funds System of the Division of Workers Compensation to run properly.

5 F. Identification of the minimum mainframe hardware and telecommunications configuration required to run the systems/sub-systems. The Office of (Information Systems, assisted by the Business Program Manager, will gather and record the following information which must be provided to GGCC: 1. A description of the mainframe, peripherals, and telecommunications components (and the required capacity) needed to support the system. (See the worksheet attached as appendix A). 2. The minimum number and location of terminals to be used for input/inquiry; include a description of the activities which will be performed from the terminals. 3. An estimate of the transaction volume, by line and drop, for the required telecommunications network. 4. The minimum frequency of execution of batch processes (or the time periods and minimum length of availability for on-line systems). G. Identification of special forms required during the execution of the systems. 1. The Office of Information Systems will identify all special forms which must be transported to the hot site to meet on-site printing requirements. OIS and the user will jointly estimate the number of forms required for the maximum 14 day hot site processing. 2. The Business Program Manager is responsible for ensuring that an adequate supply of forms is on hand, at all times, to meet the requirement for a 14 day supply for hot site processing. H. Preparation of the Disaster Recovery Plan (GGCC Failure). The Disaster Recovery Plan for this category of disaster will be in the form of a Script. The Script describes, in chronological sequence, all activities and tasks which must be executed to ensure an orderly resumption of processing (at the GGCC hot site) for designated

6 Critical Business Activities. The plan must define all recovery/operational teams and assign specific responsibility for activity/task completion. The physical preparation of the plan is the responsibility of the Office of Information Systems. I. Sample Script for Disaster Recovery Plan. SYSTEM - WORKERS COMPENSATION SPECIAL FUNDS 1. ACTIVITY: Maintain Backup Copies of Critical Automated Files. RESPONSIBILITY: OIS FILES TO BE INCLUDED: a. File a - data file (1) Frequency of backup (2) Location of backup storage b. File b - JCL file (1) Frequency of backup (2) Location of backup storage c. File c - source/object application programs (1) Frequency of backup (2) Location of backup storage 2. ACTIVITY: Declaration of a Disaster RESPONSIBILITY: General Government Computer Center (GGCC) Formal notification will be in the form of a telephone call to the Director of the Office of Information Systems. This notification will be followed by written notification providing any additional information available (estimated duration of the disruption, special plan execution requirements, etc.).

7 3. ACTIVITY: Activation of the Disaster Recovery Process within CDLE. RESPONSIBILITY: Director, Office of Information Systems TASKS: a. Notify the OIS Support Services and Communications Section managers. b. Notify the Applications Team Leader, DOWC. c. Notify the Executive Director, Department of Labor and Employment. d. Notify the Director, Division of Workers Compensation. TIME FRAME: Within 15 minutes after the declaration of a disaster by GGCC. 4. ACTIVITY: Activate the CDLE Disaster Recovery Coordination Center. RESPONSIBILITY: OIS, Manager of Support Services. TASKS: Plan. a. Coordinate and monitor all aspects of the execution of the Disaster Recovery b. Establish liaison with GGCC, including providing schedulers and other required personnel at the GGCC Control Center at 1515 Sherman. c. Provide or arrange for transportation of people and things to required locations. d. Function as a clearing house for all problems encountered. e. Establish (with GGCC) a schedule of pickup and delivery of systems input and output. TIME FRAME: Operational within 30 minutes after activation of the plan. 5. ACTIVITY: Deliver Backup Files to GGCC for transport to the hot site. RESPONSIBILITY: OIS, Support Services Manager TASKS: a. Locate tapes. b. Box for shipment. c. Deliver to GGCC.

8 TIME FRAME: Within 2 hours after activation of the plan. 6. ACTIVITY: Deliver Special Forms to GGCC. RESPONSIBILITY: OIS, Support Services Manager. TASKS: a. Locate forms. b. Box for shipment. c. Deliver to GGCC. TIME FRAME: Within 2 hours after activation of the plan. 7. ACTIVITY: Staff GGCC Control Center with Applications Programmers. RESPONSIBILITY: OIS, Applications Team Leader, DOWC. TASKS: Monitor system restoration and job-stream execution; assist schedulers and Support Services personnel in problem identification and resolution; modify applications programs and/or JCL if necessary. TIME FRAME: Operational within 4 hours after activation of the plan. 8. ACTIVITY: Move Data Input operations to the GGCC Control Center at 1515 Sherman. RESPONSIBILITY: Director, Division of Workers Compensation. TASKS: Invoke internal procedures to accomplish business activities from the GGCC Control Center. TIME FRAME: At the discretion of the Director, DOWC. The system will be available to accept input approximately 8 hours after the declaration of a disaster by GGCC. 9. ACTIVITY: Produce documents required to support manual contingency plans. RESPONSIBILITY: OIS Support Services TASKS: a.locate source documents (tapes, disks, etc.).

9 b.deliver to commercial Service Bureau for printing. C.Deliver hard copy to Business Program Manager. TIME FRAME: Within 4 normal working hours after the declaration of a disaster by GGCC. 10. ACTIVITY: Invoke manual contingency plans for non-critical business activities. RESPONSIBILITY: Director, DOWC. TASKS: Continue non-critical business activities without the support of automated systems for a period of time not to exceed 14 calendar days. TIME FRAME: Operational immediately after the declaration of a disaster by GGCC. 11. ACTIVITY: Commence planning for migration from hot site to alternate facility. RESPONSIBILITY: Director, Office of Information Systems. TASKS: Convene a task force consisting of OIS and user personnel to continually monitor the activities at the hot site as well as user executed manual contingency plans; develop plans for the re-integration of the results of manual operations into the results of hot site processing; document a plan for resumption of normal operations as soon as an alternate site is operational (not more than 14 calendar days at the hot site). TIME FRAME: Within 48 hours after the declaration of the disaster. PART V PLANNING CONSIDERATION, DEPARTMENT CONTROLLED FACILITIES FAILURE This part describes the activities which are the responsibility of the department in the planning for and response to a disaster caused by failure of facilities controlled by the department, and the failure is expected to last 3 business days or more. SUMMARY OF PLANNING ACTIVITIES DISCUSSED IN THIS PART A Identify critical tasks, components supporting these critical tasks and the amount of time the program

10 can do without the task component. B. Rank the critical tasks from most critical to least critical. C. Define the type of expertise teams needed to develop the plan based on the critical tasks ranking. - forms team - facility team - network team - pc team - personnel team D. Assign departmental personnel to the various teams based on expertise required. E. Develop a disaster recovery plan. 1. For each critical task component determine the amount of time that would normally be required to replace the task component. If the time that would normally be required to replace the task component exceeds the time the task component can be done without complete the following steps. 2. Determine the type and amount of resources/components required to complete the task 3. Determine the steps required to assure the resources to replace the task component will be available in the time stated. 4. Determinealternative processes to be completed to meet the gap between the time task is needed and the time to replace. 5. Determine the resources required to complete the alternative process 6. Determine if the required resources to complete the alternative processed are available 7. Determine the alternative processes which can be completed bases on available resources DETAILED EXPLANATION OF PLANNING ACTIVITIES

11 A. Identify critical tasks, components supporting these critical tasks and the amount of time the program can do without the task component (Logic: The tasks completed by the program must be defined to assure important tasks are covered in the plan to recovery from a disaster.) 1. Document the objectives of the program being administered example UI Benefits To pay _ -- eligible claimants the entitled amount of unemployment insurance benefits in a timely and accurate manner 2. Document the tasks required to meet the objective example UI Benefits: Take the initial claim and assure the claimant meets the monetary requirements. Issue the separation notices to the claimant and the employer. Issue the initial payorder to the claimant. Process the initial payorder through the scanner. Determine if the eligibility criteria has been met and compute the benefit amount. Print the UI benefit warrant. Sign and mail the UI benefit warrant. 3. Document the components supporting these critical tasks and the time the component can be done without. EXAMPLE: UI Benefits Take the initial claim and assure the claimant meets the monetary requirements. - UI claimstakers (trained) (24 hours) - wage history information (12 hours) - initial claim (claimant or hard copy) (24 hours) - automatic call distributor phone system (24 hours) Issue the separation notices to the claimant and the employer. - wage history information - job separation forms - CUBS application - UI Claimstakers (trained) Issue the initial payorder to the claimant. - payorder forms - CUBS application - claimant information

12 Process the initial pa6yorder through the scanner. - place to receive the completed payorders - personnel to sort the payorders (scannable and nonscannable) - scanner - scanner tapes - crt to review the scanned information on tape Determine if the eligibility criteria has been met and compute the benefit amount. - trained personnel for monetary batch processing - scanner equipment - CUBS application - trained personnel to compute the benefit amount for benefits paid manually (EUC, etc) Print the UI benefit warrant. - blank check stock - high speed printer - personnel to process the check stock (deliver warrants and receive printed warrants and forward printed warrants for signing) -CUBS application Sign and mail the UI benefit warrant. - warrant signing machine - signature plates - trained personnel - printed warrants B. Rank the critical tasks from most critical to least critical (Logic: Since all the tasks can not be completed during the recovery from a disaster, it is essential to rank the most critical tasks required to continue the program s business) EXAMPLE: UI Benefits: Process the payorder through the scanner. (payorder required as the basis for UI benefit payment) Determine if the eligibility criteria has been met and compute the benefit amount. (eligibility determination required before benefit payment can be issued) Print the UI benefit warrant. (eligible claimants need their benefits) Sign and mail the UI benefit warrant. (eligible claimants need their benefits)

13 Take the initial claim and assure the claimant meets the monetary requirements. (the short delay of not taking the initial claim can be made to work due to the waiting period required on new claims) Issue the separation notices to the claimant and the employer. Issue the initial payorder to the claimant. C. Define the type of expertise teams needed to develop the plan based on the ranking of the critical tasks (Logic: Need to sort critical task components by function to assure the tasks are assigned to the proper area.) 1. Determine the components supporting the critical tasks (see step a3 and b) EXAMPLE: UI Benefit: Process the payorder through the scanner. - place to receive the completed payorders - personnel to sort the payorders (scannable and nonscannable) - scanner - scanner tapes - crt to review the scanned information on tape Take the initial claim and assure the claimant meets the monetary requirements. - automatic call distributor phone system (24 hours) - UI claimstakers (trained) (24 hours) - wage history information (12 hours) - initial claim (claimant or hard copy) (24 hours) Print the UI benefit warrant. - blank check stock - high speed printer - personnel to process the check stock (deliver warrants and receive printed warrants and forward printed warrants for signing) - CUBS application Sign and mail the UI benefit warrant. - warrant signing machine - signature plates - trained personnel - printed warrants.

14 2. Allocate the critical tasks components into groups EXAMPLE: UI Benefit:. Hardware task components: - scanner - scanner tapes - automatic call distributor phone system (24 hours) - high speed printer - warrant signing machine - signature plates 3. Assign the critical tasks groups to teams EXAMPLE - UI Benefit:. Hardware task components assigned to Hardware Group D. Assign departmental personnel to the various teams based on expertise required (Logic: program personnel have expertise in various areas and this expertise needs to be focused where needed) 1. Define personnel with expertise in the areas needed (review the groups of critical task c o m p o n e n t s ) 2. Request experts to participate in the plan development 3. Document the participants on each team 4. Explain purpose and responsibilities to team members E. Develop a Disaster Recovery Plan 1. For each critical task component determine the amount of time that would normally be required to replace the task component. If the time that would normally be required to replace the task component exceeds the time the task component can be done without complete the following steps. (Logic: it is essential to know which of the critical task components will not be available within the time in which the critical task component can be done without.) a. Assign the task component to the team member with expertise in the area to determine the

15 amount of time to replace the task component. b. Discuss with senior program management the time frame that the critical task component can be done without. c. Document the results of the work. EXAMPLE - UI Benefits: Hardware team (selected since the most critical task component Process the payorder through the scanner involves hardware) task components: - scanner (72 hrs time to replace, 24 hrs needed within) - crt (24,24) - scanner tapes (72,24) - automatic call distributor phone system (120,24) - high speed printer (12,24) - warrant signing machine (18,24) - signature plates (18,24) 2. For the critical task components with a time criticality factor (longer time to replace under normal circumstances than a program can do without the resource), determine the type and amount of resources required to complete the task. (Logic: must know the resources involved to currently complete the task so when planning for alternative processes a baseline of resource requirements is available.) a. Review the task components and identify the task components with a time criticality. EXAMPLE - UI Benefit:. scanner (scanning the payorders into the CUBS application) b. Gather information on the process currently conducted using the time critical resource. Based on the information gathered (through conversations with the applicable program personnel) quantify the resources required. EXAMPLE - UI Benefit:. scanning process requires six personnel (5 technicians and a scanner operator

16 12 hours five days a week) c. Document the information gathered and discuss with other team members 3. Determine the steps required to assure the resources to replace the task component will be available in the time stated. (Logic: need to be sure the replacement time used as a basis for making decisions is accurate.) ANY OF THE ESTIMATED TIME REPLACEMENT FOR CRITICAL TASK COMPONENTS WHICH THE TEAM DOES NOT FEEL REASONABLY SURE OF SHOULD BE REVIEWED. a. Determine the task component with a time criticality (see 2 above) b. Review the applicable contract and other agreements and document: - vendor company - vendor contact and number - service availability c. Contact vendor and assure the stated replacement times are reasonable. Document the conversation (person talked to, date, time and conversation). 4. Determine alternative processes to be completed to meet the gap between the time the task component is needed and the time required to replace. (Logic: entire purpose of the disaster recovery plan is to develop alternative procedures for critical tasks needing to be completed until a recovery to normal business can be attained.) a. Quantify the time period that the alternative process needs to function for by comparing the time required to replace the critical component to the time the critical component can be done without. EXAMPLE - UI Benefit:. scanner 48 hours (72 hrs time to replace, 24 hrs needed within) b. Review the information on the process currently conducted using the critical task component

17 (see step 2,b). Document the current resource requirement. c. Discuss with applicable team members potential solutions for meeting the critical time period. Document the potential solutions. EXAMPLE - UI Benefit:. execute UI benefit payments to all claimants with the understanding adjustments to the claimants benefits will be made after the scanner is functioning - execute the UI benefit payments to all claimants except for the claimants with earnings (monetary batches). manually review all submitted payorders and enter the results into the CUBS application 5. Determine the resources required to complete the alternative processes.(logic: must know the resources required for the alternative process in order to determine if the alternative process can be accomplished.) a. For the alternatives developed in step ( (above) quantify the resource required by: (1) discuss with the applicable program personnel the resources requirement and document. EXAMPLE - UI Benefit:. execute UI benefit payments to all claimants with the understanding adjustments to the claimants benefits will be made after the scanner is functioning (would provide 5 technicians and a scanner operator to meet other critical needs). - execute the UI benefit payments to all claimants except for the claimants with earnings monetary batches) (would require an addition 3 technicians less the one scanner operator) (2) discuss the results with the team members and validate the resource estimates with other program personnel with expertise in the area.

18 6. Determine if the required resources to complete the alternative process are available. (Logic: a disaster recovery plan for the entire program is being generated not at the individual task component level, thus must know if the resources are available.) a. Meet with the other expert teams and determine and document the resource requirements of the entire critical task components. Summarize the total resource requirements and the available resources. 7. Determine the alternative processes which can be completed based on available resources. (Logic: not all alternative processes for critical task components will be able to be implemented, so management of the program must use input from the teams of experts and their own professional judgement to determine to determine the alternative processes to fund.) a. Present the work of the teams of experts to the program management and thorough discuss the alternative processes and the logic for the conclusions reached. b. Document the conclusions reached and the logic for the conclusions. PART VI GENERIC PLANNING CONSIDERATIONS A. Identification of Documents Required to Support Manual Contingency Operations. Nearly all work processes in the department rely, to some degree, on support from automated systems. In many instances the implementation of manual procedures will require that some record of the current state of an automated process be available to use as the starting point for the manual process. EXAMPLE: The work process of scheduling hearings is supported by an automated system. In the event of a disaster this work process will be executed manually. However, in order to manually administer the hearings scheduling process, the most current schedule of hearings activities must be available immediately after the automated

19 process becomes inactive. 1. The Business Program Manager must identify all instances in which the use of a product from the existing automated system is a prerequisite to beginning the manual alternate work process. A full description of all such items must be provided to the office of information systems during the preparation of the Disaster Recovery Plan. 2. OIS must ensure that existing job streams are modified to generate (for off-site backup storage) the necessary products (i.e., a tape copy of a print file which contains a current master list) to be used to produce the required documents. B. Planning for Other Than Work Processes. The Disaster Recovery Plan must address several factors which do not necessarily apply directly to the work processes. These factors may or may not be applicable to any given disaster. It is necessary for the Business Program Manager to recognize the potential for these factors to be significant in any disaster, and to assign to specific teams or individuals the responsibility to execute the tasks associated with these activities. Some of these generic planning considerations might be addressed at the Division or Business Program level, while others might be addressed at the site or geographical location level. It is the responsibility of the Division or Office Director to make a determination which factors will be addressed at what level. It may, in some cases, be desirable to involve the Buildings and Facilities unit in the planning process and to assign them the responsibility for certain activities. The activities and tasks to be included in the Generic portion of the Disaster Recovery Plan are as follows: 1. Establish a Public Relations Center. - Issue Press releases.

20 - Answer inquiries from clients, media, and other state agencies. 2. Establish a Control/Coordination Center. - Coordinate all business activities of the Disaster Recovery Effort. 3. Establish Security for Damaged Facilities. - Prevent unauthorized entry (curiosity seekers, looters, etc.) into damaged premises. 4. Conduct a Damage Assessment. - Coordinate with local authorities to determine when premises can be safely inspected. - Coordinate with insurance carrier and local authorities to conduct a physical inspection. 5. Develop Cost Estimates and Establish Procurement Procedures. - - Estimate costs to replace equipment and premises. - Develop emergency procurement procedures which will ensure expeditious processing of urgent procurement activities. 6. Obtain Alternate Site Space. - Re-establish operations at an interim or a permanent alternate physical facility. - Restore or re-route communications lines to alternate site. - Restore or re-route other utilities. 7. Address the needs of employees. - Assistance to the families of casualties. + Insurance + Relocation - Financial Assistance + Arrangements for payment of salary if payroll system is affected 8. Build or augment work staff.

21 - Identify and transfer trained replacements. - Hire and train new employees.