Alabama Supercomputer Center Alabama Research and Education Network Tornadoes Disaster Continuity of Operations Experience 1
Success Story Alabama Supercomputer Center operated continuously through the April 27 th tornadoes disaster and week following Maintained full generator power for seven days except for 40 minutes on UPS All data center services were maintained during this time Critical network backbone and Internet services were available continuously Additional IT services for recovery support were supplied to State agencies (e.g. AEMA), City of Huntsville, school systems, universities, community colleges, and FEMA Lessons learned are being incorporated into operations procedures 2
Statewide Education Network NLR Internet2 Internet Universities K-12 Alabama Supercomputer Center State Agencies Support Community Colleges Public Libraries ASA s high speed network is connected to the Supercomputer Center, Internet, Internet2, & NLR 3
Alabama Supercomputer Center Dual, redundant UPS systems Dual fuel, parallel, redundant generators Loop 12,400 V utility feeds Full building electrical plant is handled by generator systems Three fiber entry paths Facility Improvements and Modernization of Previous Five Years Enabled Continuous Operation During Week Long Power Outage 4
Network Backbone Infrastructure ASA s network backbone design provides redundant 10G transport among all major network hubs Internet gateway access in Atlanta and Dallas with failover capacity 5
Network Backbone Has Vendor and Path Diversity ASA s new network backbone design mitigates the risk of having all Internet and Internet2 connections through one point (previously Atlanta) 6
Client Endpoints Service More than 100 client locations lost network service Some telecommunications outages (bulk in Marshall and Dekalb counties) Most due to loss of power Six schools destroyed More risk because redundant connections to endpoints are expensive even if a diverse option is available 7
Staff Was Key to Maintaining Operations Exemplary staff performance was central to maintaining 24x7 data center operations Some employees worked around the clock in first 48 hours Special plan for 12 hour shifts for curfew compliance Staff conducted some Tier 2 and Tier 3 support off-site Disaster planning was modified and implemented during the disaster recovery Key servers were moved from Huntsville to Montgomery on Sunday, May 1 st to ensure full Internet and client email/web services would be maintained in case of second generator failure Strong working relationship and excellent communications between ASA and professional services contractor, CSC Enabled quick responses to clients immediate needs without formal approvals Worked jointly to keep facility, data center, statewide network, applications operational 8
Special Support to Clients City of Huntsville Data Center Worked with API Digital to provide emergency Internet access to the City of Huntsville Provisioned a 1Gbps Ethernet connection with a block of usable address space and API used this to configure the City's Internet access during the recovery Federal Emergency Management Agency (FEMA) Established 50 Mbps VLAN and provided Singlemode Fiber SFP optics to the Fire College to support EMA for FEMA to setup a command center in Tuscaloosa at the Alabama Fire College node on AREN. New muliple T1s connectivity expedited for Joint Field Office (AEMA/FEMA) in Birmingham (upgrade to Metro Ethernet service ordered) Alabama Emergency Management Agency (AEMA) Increased EMA HQ in Clanton to 50 Mbps from 10 Mbps normal service Established service to new Birmingham field site for AEMA/FEMA Auburn University Backup Internet 9
Special Support to Clients George C. Wallace Hanceville Community College Hosted the GCW Hanceville website, Student Blackboard server, and online registration servers due to storm damage at campus Their planned DR site, Calhoun Community College, was also down due to the storm and unavailable Tuscaloosa City Schools Assisted The University of Alabama in setting up a wireless link to Tuscaloosa City elementary school that used online textbooks exclusively Madison County Schools Hosting their website because their server had no power or Internet Alabama A&M University Provided loaner firewall equipment to AAMU Alabama Department of Education Provided daily updates on status of school systems Internet connectivity 10
Lessons Learned Summary AREN network design performed flawlessly 10 Gbps backbone (installation completed in late 2010) kept all network functions intact However, loss of local carriers needs to be a contingency in DR plan ASA provided fuel source to API to keep running local network hub operational API Digital is a major offsite location that connects AREN WAN connections Loss of API site would have removed redundancy for the Huntsville ASC data center New sources/options established for fuel for generator More attention to complete loss of facility needed Expect that full preparations for disasters will never be enough All disaster scenarios will not be forecast in a plan No two disasters are alike Disaster planning generally did not account for a scenario for losing all power in North Alabama Lessons learned are being incorporated into operations procedures 11
Alabama Supercomputer Authority State of Alabama Leader and Trusted Partner for Technology 12