Title: Georgia Enterprise Technology Services Disaster Recovery Category: Disaster Recovery and Business Continuity Readiness Contact: Mr.

Transcription

1 NASCIO 2015 State IT Recognition Awards Title: Georgia Enterprise Technology Services Disaster Recovery Category: Disaster Recovery and Business Continuity Readiness Contact: Mr. Calvin Rhodes CIO, State of Georgia Executive Director, GTA State: Georgia Project Initiation Date: March 2010 Project Completion Date: August 2014

2 EXECUTIVE SUMMARY A large component of the Georgia Enterprise Technology Services (GETS) program, a groundbreaking initiative for the Georgia state government, was to address the huge risks found by both internal and independent assessments in the area of disaster recovery (DR). These risks were significant and included aging equipment, underfunding, inconsistent operating standards and failure to comply with industry best practices. In addition, the state lacked the latest technical skills. Georgia turned to IBM, a private-sector leader in technology service delivery, to correct these problems. Together, they established the state s first comprehensive technology DR program. As a result, the state of Georgia is saving money, staying current with modern technology, ensuring confidential data remains safe and secure, and protecting business continuity from outside threats. The path toward IT transformation was not easy, but it was well worth the effort. The changes that have already taken place and are continuing to evolve will benefit all Georgians well into the future. BUSINESS PROBLEM AND SOLUTION DESCRIPTION The state of Georgia was exposed to significant risk by not having adequate DR plans. The Georgia Technology Authority (GTA) recognized the need to develop and implement a plan to protect critical systems in the event of a disaster. Many types of unplanned events threaten the functionality of state agency IT systems, affecting the availability of services and data these systems provide. Georgia s state agencies faced momentous challenges in addressing these concerns, including difficulties with logistics and repetitive costs. Data and critical applications were dispersed across multiple data centers, on unrelated platforms, relying on dissimilar data protection strategies and procedures. The state lacked the appropriate location and facilities required for a stable and secure recovery site. GETS Participating Agencies Department of Community Health (DCH) (DDS) Department of Human Services (DHS) Department of Natural Resources (DNR) Department of Administrative Services (DOAS) Department of Revenue (DOR) (GBI) Georgia Department of Corrections (GDC) Georgia Technology Authority (GTA) State Accounting Office (SAO) Department of Juvenile Justice (DJJ) Department of Public Health (DPH) Office of Planning and Budget (OPB) Department of Behavioral Health and Developmental Disabilities (DBHDD) 1

3 To address these challenges, GTA partnered with IBM to develop and deliver an evolutionary DR services plan that would meet the needs of 14 GETS agencies, each with its own unique recovery scenarios and requirements for more than 400 applications. For everyday operations, GTA and IBM developed a plan to consolidate applications into a new, common hosting environment: Georgia Enterprise Technology Services Hosting Environment (GETS-HE). The new hosting environment takes advantage of economies of scale, including common infrastructure, platforms, and data protection platforms and procedures. These same economies of scale could then be applied to a common DR services plan. A four-tiered recovery scheme was developed to identify critical systems and applications, as well as how rapidly each system must be recovered in the event of a disaster. Tier-0 designates the most critical systems that need to be recovered within the first 32 hours of a disaster declaration. Tier-1 applications are recovered within one week. Tier-2 applications are recovered within two weeks. Tier-3 applications are recovered at the agency s discretion, if and when the agency determines a need based on the severity and duration of the disaster. State of Georgia Tier-0 Applications Agency Name Department of Community Health Department of Community Health Application Name Computerized Criminal History Law Enforcement Message Switch Laboratory Information Management System Justice Trax Investigative Case Management System Membership Enrollment Management System Citrix Presentation Server Suite Interface Modules Centralized License Processing GBI Motor Vehicle Records Driver s License System 2

4 As servers were migrated into the GETS-HE, they were converted when possible to a virtual platform, which was established to support security and regulatory data classifications. A common storage area network (SAN), managed under SAN Volume Controllers (SVC), allowed for implementing virtual data segregation and global mirroring to provide real-time data replication to a remote site. As applications designated Tier-0 were migrated into the GETS-HE, storage requirements were satisfied through the use of the SAN, which enabled data to be replicated at the remote recovery site. IBM Tivoli Storage Manager (TSM) was selected for the common data protection platform used to back up all applications migrating into the GETS-HE, including Tier-1, Tier-2 and Tier-3. Backups were scheduled to meet each agency s recovery objectives. Backups support both production and DR by vaulting copies off-site, so they can be shipped to the recovery site in the event of a disaster. Disaster Recovery Plan Facts Over 400 applications have Disaster Recovery coverage Complete network can be built out and functional in first eight hours of DR event Mainframe is up within first 18 hours of DR event Critical host and server-based applications available within 32 hours The IBM Resiliency Services location in Boulder, Colorado, was selected for the remote DR recovery site. A specific state of Georgia Rapid Recovery SAN environment for storing replicated data is maintained onsite. Economies of scale apply to the recovery site, where shared hardware is maintained, including mainframe services and server hardware that match or exceed production specifications. Taking advantage of the shared hardware at the recovery site, systems are pre-cabled to SAN storage. SVC virtual disks are also predefined, but as mainframe, host and server hardware are selected they only need to be patched in at the time a disaster is declared. This preparatory step, in combination with the replicated operating system and synchronized data, allows for the mainframe to be recovered within 18 hours and Tier-0 host and server-based applications to be available within the designated 32-hour recovery objective. SAN storage using data synchronization also achieves the same recovery objective. At the time a disaster is declared, the backup tapes will be identified, retrieved and validated. Once the TSM environment and other data-backup software are restored and server hardware platforms provisioned at the recovery site, Tier-1 applications will be restored from tape, validated and made ready for acceptance processing by the appropriate agencies. Restoration and acceptance processing begin within the first 50 hours of a disaster and continue for up to one week, depending on the complexity of the application and the number of hosts or servers involved. During a disaster, agencies will determine whether to recover Tier-2 and Tier-3 applications based on the prevailing status of the disaster. If determined necessary, these applications will be recovered using a tape-based recovery strategy. 3

5 SIGNIFICANCE The primary focus of the DR services plan is to support applications hosted in the GETS-HE. However, server and application migrations to the GETS-HE are a lengthy process, so a DR services plan was needed to support applications both before and after migrations are completed. The DR services plan, which supports more than 400 applications, and the recovery site had to be versatile and flexible to support an application s current status and location, until it was fully migrated into the GETS-HE. IBM performs annual DR exercises for the GETS environment; four have been successfully completed. Participating agencies set unique criteria for their specific DR plan. The agencies score the exercise independently, and the results are bound by service-level agreements with GTA. Multiple data centers have to be simulated inside the recovery site, until all the servers and applications are eventually migrated into the GETS-HE. Simulations are necessary to isolate the DR testing from the production network, while still allowing access and data exchange with the production environment. Over five years, DR test plans and objectives have changed. This speaks to the evolution of service delivery processes required in a developing environment. Yet, changing regulatory requirements actually helped create more stability by leading to standardized hardware platforms, which in turn reduced variations in operating systems. Architecture (based on the GETS-HE production network) is implemented at the recovery site to match connectivity and security postures. A portion of the network is part of the shared services of the recovery facility. All shared server and host hardware are pre-cabled into the shared network infrastructure, so logical network configurations can rapidly bring networks into operation. Permanent network components are also deployed in Boulder. These include switches, routers, firewalls and network infrastructure services. The permanent components are integrated with the shared recovery site infrastructure, so the complete network and associated connectivity are functional and available within the first eight hours of a disaster. 4

6 BENEFITS OF THE PROJECT Benefits realized through the DR environment created by the GTA and IBM partnership include: A greater assurance that critical data and services can be re-established for Georgia residents in the event of a disaster. DR requirements are reduced when servicing standard server hardware and operating systems to implement preparatory actions that shorten system setup time and result in applications being restored much more rapidly. The increased flexibility in the DR environment allows for new applications to be added more quickly to the DR program. State agencies are better able to meet regulatory requirements associated with applications in the GETS HE. Governance of the DR services plan takes place without distracting agencies from their daily business functions. Global mirroring of critical data allows applications to be recovered much more quickly than employing a tape-based recovery strategy. More efficient use of resources, through economies of scale, allows a greater number of applications to be serviced in the DR environment. Through its partnership with IBM, the state of Georgia has access to established facilities, a wealth of knowledge in DR methodologies, distributed local access facilities providing direct connectivity into a secure recovery site, and an extensive depth of experienced personnel all of which can be applied to recovery efforts in the midst of the chaos associated with a disaster. 5