Hawaii Behavioral Health Information Technology Contingency Plan Version: 1.0 Carla Gross Chief Operating Officer Prepared by: Michael Lukson
Hawaii Behavioral Health INFORMATION TECHNOLOGY CONTINGENCY PLAN Version: 1.0 This Hawaii Behavioral Health (HBH) Contingency Plan establishes procedures used to recover HBH Information and Technology Systems following a disruption in service or a critical failure event. The following objectives have been established for this plan: Maximize the effectiveness of contingency operations through an established plan that consists of the following phases: Notification/Activation phase to detect and assess damage and to activate a plan to restore and recover affected systems. Recovery phase to restore temporary IT operations to support continued operations until the reconstitution phase is complete. Reconstitution phase to restore/rebuild the affected IT systems back to normal operations. Prepared by: Michael Lukson Date: 05/01/15 Reviewed by: Jessica Wong-Sumida Date: 05/08/15 Approved by: Carla Gross Date: 05/15/15
1.1 BACKGROUND This HBH System Contingency Plan has been developed as required under the Office of Management and Budget (OMB) Circular A-130, Management of Federal Information Resources, Appendix III, November 2000, and the Health Insurance Portability and Accountability Act (HIPAA) Final Security Rule, Section 164.308(a) (7), which requires the establishment and implementation of procedures for responding to events that damage systems containing electronic protected health information. This HBH System Contingency Plan is promulgated under the legislative requirements set forth in the Federal Information Security Management Act (FISMA) of 2002 and the guidelines established by the National Institute of Standards and Technology (NIST) Special Publication (SP) 800-34, titled "Contingency Planning Guide for Information Technology Systems" dated June 2002. 1.2 APPLICABILITY The HBH System Contingency Plan applies to HBH Information Technology systems and delineates the necessary procedures and resources required in the event of a systems failure. The goal of these procedures is to restore normal IT operations and business functionality. The HBH System Contingency Plan applies to the Operations Division (OPDIV) and all other persons associated with HBH Systems as identified under Section 2.3 Responsibilities. The HBH Systems Contingency Plans is supported by the HBH Recovery and Backup Plan. This plan lays out the specific backup operations used to protect HBH critical data. It also lays out the specifics of critical HBH systems, where they are physically located, and the general methods for backup. The HBH Systems Contingency Plan is also supported by the HBH Technology Plan which lays out expected technology related updates or reviews on current systems. 1.3 SCOPE 1.3.1 Planning Principles This plan is designed for two basic scenarios. a. SINGLE SYSTEM FAILURE: The first is a single critical system failure. This would be a single system which failed due to software or faulty hardware. An example would be a failure of the file server in Honolulu. This type of event would cause the temporary loss of IT resources for the type of system failed. In this example a file server failure in Honolulu
would deny OPDIV staff the use of file server data such as user and workgroup folders. However, other operations such as email, billings and information management which do not rely on the failed system would still be functional. b. SITE CRITICAL SYSTEM FAILURE: This would be a system failure at a specific location due to a disaster event. Examples of this type of even would be loss of a critical location such as the Honolulu office to possibly due to a natural disaster event such as a Hurricane, Tsunami or building fire. This type of event would cause major system outages and affect business operations more seriously than a single critical systems failure. 1.3.2 Contingency Plan Action In the event of a failure of an HBH critical systems failure contingency operations will be put into place. These operations will have two or three phases. Phases Notification/Activation Recovery Reconstitution Phase Overview The notification/activation phase is the determination after a critical system failure of which plan type to implement and the notification of HBH system users of the system outage and expected recovery estimate. Recovery is the temporary restoration of an HBH system assets. Depending on the specific system this is not always possible. For instance in the failure of the email a temporary restore would not be possible. Only the reconstitution of the email server would solve this system outage. However, some systems as the file server user and work groups data files can be restored for temporary usage while system reconstitution operations are occurring. This phase is rebuild of the affected system back to its normal state. The two types of plan operations are: a. Quick Recovery Operations: For system failures where the expected recovery time is estimated to be under 48 hours, the determining authority may decide to not implement a recovery phase and accept the service loss until system reconstitution. b. Full Recovery Operations: In full recovery operations scenario it has been determined that full system reconstitution will take longer than 48 hours. In this plan the determining authority has decide to allocate resources to the recovery phase to try reduce the system impact on normal business operations.
A determination will be made by the Chief Operating Officer which type of recovery plan will be put into place. This decision will be made depending on the type of system which has failed. Certain systems such as the domain controller, and email do not the ability to have a recovery (i.e. temporary) system put into place. Other systems such as the file server can have a recovery system put into place for business operations. 1.4 REFERENCES/REQUIREMENTS This HBH Contingency Plan complies with the OPDIV IT Contingency Planning Policy as follows: The organization shall develop a contingency planning capability to meet the needs of critical supporting operations in the event of a disruption extending beyond 48 hours. The procedures for execution of such a capability shall be documented in a formal contingency plan and shall be reviewed at least annually and updated as necessary. Personnel responsible for target systems shall be trained to execute contingency procedures. The plan, recovery capabilities, and personnel shall be tested to identify weaknesses of the capability at least annually. The HBH System Contingency Plan also complies with the following federal and departmental policies: The Computer Security Act of 1987 OMB Circular A-130, Management of Federal Information Resources, Appendix III, November 2000 Federal Preparedness Circular (FPC) 65, Federal Executive Branch Continuity of Operations, July 1999 Presidential Decision Directive (PDD) 67, Enduring Constitutional Government and Continuity of Government Operations, October 1998 PDD 63, Critical Infrastructure Protection, May 1998 Federal Emergency Management Agency (FEMA), The Federal Response Plan (FRP), April 1999 Defense Authorization Act (Public Law 106-398), Title X, Subtitle G, Government Information Security Reform, October 30, 2000
2 CONTINGENCY OPERATIONS 2.1 SYSTEM DESCRIPTION AND ARCHITECTURE A. HARDWARE OVERVIEW: In the event of a critical systems failure one consideration is that the organizational hardware is not off the shelf items which can be replaced locally. Rack mounted servers and network switches will have to be ordered from specialty companies and delivery of replacement systems may take longer than a week. Because each HBH has a firewall appliance with a VPN tunnel a temporary restore may be possible using existing domain servers or file servers via this method. B. DOMAIN CONTROLLER SYSTEM: HBH has two domain controllers located in Honolulu and Hilo. These domain controllers provide logon and file security authentication for HBH sites. In the event of a critical failure in one facility the domain controller should be able to provide logon and file authentication using VPN access until a new system is rebuilt. In the event of a system failure on the domain controller, a general overview of what should occur: 1. New rank mounted server hardware acquired. 2. Windows Server Operating System installed and setup for HIBH.org domain. 3. A restoration of HIBH.ORG domain occurs from the system backup file. C. MANAGEMENT INFORMATION SYSTEMS (NPA WORKS): The HBH management and information systems is called NPAWorks. This system is provided by CodeMetro and physically located in California. Work performed. Because this is a remote web based application a site critical failure should not impact the use of this system. The use of any internet capable workstation should allow HBH to continue using this system. D. BILLING SYSTEMS: Billing is performed via NPAWorks and the Department of Education provider data system. Both of these systems are maintained by external agencies. A critical system failure at an HBH location will not affect billing output. The use of any internet capable workstation should allow HBH to continue billing operations.
E. ACCOUNTING SYSTEM: The HBH Accounting system is Quickbooks. In the event of a critical failure the following primary events will have to occur to bring the system back online: 1. CLIENT WORKSTATION INSTALL: A reinstall of the accounting system on the client. The client system will also have to be activated. 2. SERVER CLIENT INSTALL: A reinstall of the accounting system (server) will have to occur. 3. DATA FILE RESTORED: The backup copy of the accounting system will have to be copied back to the server client location and then referenced as the data file by the client server. F. PAYROLL SYSTEM: HBH uses ADP Webpay as the system for employee payroll. In the event of a facility site failure the payroll system should not be affected. However, on the individual payroll workstations webpay users will have to install a security certificate before being able to use the system. ADP has a technical support hotline to assist with this. G. HBH EMAIL SYSTEM: HBH currently uses Microsoft Exchange as the HBH email system. This system located in the Hilo office. Email backups occur and offsite copies occur using the VPN connection between offices. In the event of failure the following general operations will occur. 1. SERVER INSTALLATION: Microsoft Exchange will have to be installed on the new server. 2. RESTORE OF DATA: The exchange data files will be restored using the system backup file. H. HBH WEB SERVER: The HBH Web server is currently located in Hilo. Backups of the website occur and are stored offsite in Honolulu. In the event of failure the following general operations will occur. 1. SERVER INSTALLATION: The designated server will have to have the correct web server software and content management system (CMS) installed. 2. WEB SITE RESTORE: The web site files should be restored from the backups. I. PRIMARY DOCUMENT STORAGE: This is the central file server where user and workgroup (i.e. Masterdocs, Oahudocs) are located.
There are two HBH file servers. One located in Honolulu and the other in Hilo. In the event of a critical system failure the following general operations will occur: 1. FILE SERVER SYSTEM REBUILD: The file server will be rebuilt and then tied into the windows domain using active directory. 2. WORKGROUP AND USER FOLDERS RESTORED: The workgroup and user files will be restored using backups. 3. PERMISSIONS REBUILT: The permissions will be rebuilt using active directory to protect access to the data loaded. J. PHYSICAL LOCATION OF EQUIPMENT: There are server locations for HBH. There is a locked server storage area in Hilo. The second server storage area is located in the Chief Operations Officer s office. Both server location areas are rack mounted systems and kept in secure with limited access. K. EQUIPMENT DIAGRAM VPN Connection Honolulu Site Hilo Site L. Honolulu Microsoft Windows Server Firewall VPN Firewall Hilo Microsoft Windows Server Microsoft Exchange Server Honolulu File Server Hilo File Server Hilo Web Server INTERNET Peer-to-Peer Workgroup Kauai Site Peer to Peer Workgroup Firewall Firewall Maui Site Peer to Peer Workgroup
L. External Partnerships: HBH currently has information and technology trained staff. HBH has designated employees that can add users to the domain and create emails accounts. All other technical support issues including system backups, maintenance and computer workstation setup is done using external partnerships. HBH ensures that a good synergy with external partners is established. External partners must be able to support organizations critical periods and work with management to achieve cost effective IT solutions. 2.2 LINE OF SUCCESSION In the event of a site system critical event a decision making authority will have to ensure that the contingency plan is activated. The Chief Operations Officer (COO) is the designated authority to make these decision and set the plan in motion. The plan members for contingency operations are listed in Appendix A. Plan members are responsible for reporting status information to the COO and ensuring that recover and reconstitution operations occur in a timely fashion. Any developments which affect these operations should be communicated. 2.3 RESPONSIBILITIES In the event of a facility system critical event a response team will be created using the contacts listed in Appendix A. Most system issues will be geographically centered and only the contact point listed for that region need be involved. The other team members will normally be the designating authority (COO) and the external partner (tech consultant). Because HBH has limited technical resources the majority of the work responsibility will be placed upon the external partner. With this type of setup it will be important to establish a good relationship with the external partner. The external partner should be able to commit to devoting adequate scheduling time to support system critical events when needed. Other island contact members should be able to work with the external partner to assist when needed. 2.4 TESTING AND MAINTENANCE Testing and maintenance of HBH IT systems should occur on a regular basis. Any irregularities noted in the daily system operations should be communicated to the external partner. Testing of the system backups should also occur on a frequent basis. These procedures are discussed the HBH Recovery and Backup Operations Policy.
3 NOTIFICATION AND ACTIVATION PHASE Upon the failure of a critical system actions should be taken to assess the damage to the system. The regional center point of contact individuals should discuss system outages with the determining authority (COO). The COO should contact the external partner to review the damage to the system and determine what steps need to be taken to bring the system back to operational status. 4 RECOVERY OPERATIONS Recovery operations will occur, when eligible, for systems when it has been determined that the outage will be over 48 hours. Not all systems are eligible for a temporary recovery situation. As an example the failure of the email system would not be eligible for a temporary recovery operation. The external partner and island contact representatives will be responsible for the planning and execution of recovery operations. 5 RECONSTITUTION OPERATIONS Reconstitution Operations is the restoration of the affected system back to normal usage. Once the affected system has been put back into normal working order any recovery systems should be brought down. It is important to ensure that new data or information created on the recovery systems is brought over to the newly reconstituted system. This will ensure that no work efforts have been lost. 5.1 ORIGINAL OR NEW SITE RESTORATION When new systems have been reconstituted they may be located in a new facility. It placed in a new location it is important that ventilation, security and electrical needs are met for the new equipment. If ventilation and electrical needs are not met it can affect the new equipment in an adverse manner. The external constant should be involved with the new system layout. 5.2 CONCURRENT PROCESSING In the event of a critical system failure at a facility location many normal business operations can continue with minimal delays. The following Billing and Operations The billing systems and management information systems are hosted by companies external to HBH. A facility site failure will have minimal effect on these operations. The recovery team news
to produce normal computer workstation with access to the internet for these operations to continue. Accounting Operations For accounting systems Quickbooks 2013 data file is normally hosted on a server. For recovery operations a temporary system could be put into place. First a workstation would have to be put in place with Quickbooks 2013 installed. The data file (accounting information) could be directly hosted on the workstation to support accounting operations. Payroll Operations Payroll operations use ADP webpay for payroll. Because this system is external to HBH facility critical failures will have minimal impact. To continue payroll operations you will need a computer workstation. A special security certificate will have to be installed for the payroll user. ADP can support this installation via their technical support. File Server Operations HBH uses a number of regular word processing and spreadsheet documents to support business operations. During the file server reconstitution phase a recovery file server can be put into place with the backs being located on a recovery system. This will support normal business operations. 5.3 PLAN DEACTIVATION Once new systems have been put into place. The contingency operation is coming to a close. The contingency team should evaluate what may have caused the site critical failure event. Any lessons which can be learned from the event should be documented and reported to the management team. All recovery systems should be deactivated and disassembled. It is important that data from any recovery systems be brought over to the new systems so no work is lost. Recovery systems deactivated should be protected as there may be HIPAA related information stored on them. All disk media should be destroyed in accordance with normal IT policies.
Appendix A: Contact List # Name Contact List Plan Assignment Office Phone Cell Phone EMAIL 1 Carla Gross COO cgross@hibh.org 2 Mark Chun External Partner (808) 557-1300 mchun@pacificprotech.com 3 Lorna Pedro- Villaneuva Big Island Contact lorna@hibh.org 4 Maila Kaneaiakala Honolulu Contact ronderom@hibh.org 5 Mary Santos Kauai Contact santosm@hibh.org 6 Courtney Azoulay Maui Contact azoulayc@hibh.org