Small School Disaster Recovery Plan

Transcription

1 SUNGARD SUMMIT 2007 sungardsummit.com 1 Small School Disaster Recovery Plan Presented by: Jing Ding Lourdes College 6832 Convent Boulevard Sylvania, OH March 21, :30am 9:30am Banner Technical Course ID 767 A Community of Learning

2 SUNGARD SUMMIT 2007 sungardsummit.com 2 Session Rules of Etiquette Please turn off your cell phone/pager If you must leave the session early, please do so as discreetly as possible Please avoid side conversation during the session Thank you for your cooperation! A Community of Learning

3 Introduction As a small school uses banner products as our daily administrative tools, when we are considering our Disaster Recovery (DR) plan, budget is a main issue. We tried to get a DR plan which provides maximum availabilities of banner system to our users with minimum costs. Attendees may get some ideas from this presentation for their own DR plans. 3

4 Outlines Main Issues on Disaster Recovery (DR) for Small Schools Pre-Disaster Planning DR Plan Development DR Plan Testing Implement the Completed DR Plan 4

5 SUNGARD SUMMIT 2007 sungardsummit.com 5 Topic #1 Main issues on disaster recovery (DR) for small schools A Community of Learning

6 Declaration of Disaster Recovery: Unexpected interruptions upon our daily business process. Allow essential management function to continue. Focus on the recovery of servers. Course ID 767 6

7 Main Issues Budget, budget and budget? What are your mission critical tasks for recovery? Maximum time range of data loss that you can tolerate? Course ID 767 7

8 SUNGARD SUMMIT 2007 sungardsummit.com 8 Topic #2 Pre-Disaster Planning A Community of Learning

9 Nobody wants disaster, BUT Course ID 767 9

10 We have to face it if it happens Course ID

11 Your DR Plan Your damaged data Your recovered data Course ID

12 A Solid DR Plan Starts With A Solid Base Completed DR Plan DR Plan Development DR Plan Testing Risk Analysis vcvc Rish Assessment Business fdfd Impact Analysis Recovery Strategies Course ID

13 Self-Asked Questions Spend money to rent remote servers from a professional storage company? -- We tried but eventually gave up because it was too expensive. What s the comfort level for your DR plan? -- Your comfort level is impacted greatly by your DR budget. Course ID

14 The Best Things We Can Do Within Budget Purchase necessary software and hardware. Mirror banner database server and Oracle form server - put them into physically separate locations. Image critical servers and some select workstations. Check backup (hot & Cold) processes periodically. Maintain, update and secure adequate backup materials. Course ID

15 SUNGARD SUMMIT 2007 sungardsummit.com 15 Topic #3 DR Plan Development A Community of Learning

16 Specify Purpose Your plan should then be completed and up-to-date to achieve this purpose. Our DR Plan Objective: Specify a DR team. (including alternative persons and necessary contact information) Respond to a disaster occurrence by determining the extent of the disaster. Detailed plan of actions that can be followed and implemented to guarantee the minimal interruption to our daily operation. Course ID

17 Specify Team Members First responders Secondary crisis management team keep a phone directory on hand Employee Position Office Location Office Phone Home Phone Cell Phone John xxxx Network Analyst SCH XXX XXXX XXXX Jing Ding DBA MOT XXX XXXX XXXX Mike xxxx System Administrator MOT XXX XXXX XXXX Mike xxxx VP of Finance & Administration LH XXX XXXX XXXX Laurie xxx Director of Administrative system MOT XXX XXXX XXXX Course ID

18 Specify Level of Priorities a. Servers Mission Critical Servers with high priority Non-mission critical servers with low priority b. Some select workstations May have their processes that directly impact one or more mission critical servers. Should have first priority to be recovered. Course ID

19 Example of Our Selected Critical Workstations John xxxx Liz xxxx Jing Ding Mike xxxx Michelle xxxx Teri xxxx Kathy xxxx Network Analyst Web Content Administrator DBA System Administrator Registrar (Schedule 25) Bank Reconciliation Payroll processing Course ID

20 General Procedures Maintaining and updating the disaster recovery plan. Ensuring that all DR team members are aware of their responsibilities in case of a disaster. Ensuring that hot backups and cold backups are taken appropriately. Ensuring that periodic scheduled rotation of backup media is being followed for the storage facilities. Course ID

21 General Procedures, continued. Maintaining the primary banner database server and the form server in the server room. Maintaining the mirroring of primary servers in a physical remote location. Ensuring that fire protection and UPS systems are functioning properly and that they are checked periodically. Ensuring that proper temperatures are maintained in server rooms. Course ID

22 Software 11 Copies of Symantec Backup Exec System Recovery Server. Clients configured to take a snapshot based on a 100 megabyte change to disk space on server plus or minus. Snapshots consolidated to single set daily at Midnight. Client software will be installed on all mission critical servers. Make copies of all original client software, keep record of the product key# and serial# in case for re-installation. Store and secure software copies and original software in separate places. Course ID

23 Hardware 2 Terabyte Cybernetics misan Storage Server. This server will be located in a remote location on campus from the server farm. Images of mission critical servers will be stored on the this server. Course ID

24 Recovery Strategies Hierarchy Contains Physical Standby Database Switch to the Mirroring Server Failed Recover from a full image Failed Rebuild server from Scratch Apply Apply Drive snapshot differential backup Restore database from Hot backups OS O R A C L E B A N N E R Other Software Failed Restore database from cold backups Course ID

25 Strategy #1: Switch to Mirroring Severs Offer: Physical standby database, a transactionally consistent copy of the primary database. All primary database transactions are sent to standby database simultaneously to achieve the maximum availability. The operation system, Oracle, banner and other application software are kept identical with those on the primary servers. Expectation: The fastest way to ensure the minimum downtime. Theoretically, several minutes. Course ID

26 Strategy #2: Recover From A Full Image Offer: Fully image critical servers each time a major banner upgrade has been applied or a new software is installed. The first choice for recovery in a disaster scenario that both primary servers and mirroring servers are not available. Saves time to re-install OS, ORACLE, BANNER and other application software, re-apply upgrades and patches. Based on the differential backups created from drive snapshot, applies those changes since last full image. Expectation: A fast way to enable daily operation to survive disasters and data corruptions. Course ID

27 Course ID

28 Course ID

29 Index file Basic ImageIncremental Index file Course ID

30 Strategy #3: Rebuild Server From Scratch (No! Never!) Offer: Addresses the scenario of complete loss. Re-install OS, ORACLE, BANNER and other applicable software. Rely on hot backup or cold backup to do complete recovery or incomplete recovery. Expectation: Slow, but the only way to restore and recover critical data in the worst case scenario. May have lost some data. Course ID

31 Write Down Your Plan of Action In Details Don t only rely on key persons for Disaster Recovery in case one of them can t be reached immediately. Write down your step by step recovery procedures to get ready for a scenario you need a person w/o training to do the recovery. Perform comprehensive tests of your detailed plan of actions. You may stay in the loop of development and testing until you get a completed DR plan. Your completed DR plan then needs to be tested and updated at least once a year. Course ID

32 SUNGARD SUMMIT 2007 sungardsummit.com 32 Topic #4 DR Plan Testing A Community of Learning

33 Comprehensive Tests For Each Recovery Strategy Performs scheduled tests. Knows individual responsibilities. Confirms each procedure works. Identifies the areas requiring modification. Increases confidence. Course ID

34 Testing Process Tested restore and recovery from hot backup: -- Recovery successful, about 1.5hrs. -- Confirmed that all changes made during hot backup process can also be recovered. -- planning to schedule hot backup to run every 2 hrs. Experienced restore and recovery PROD from cold backup (04/25/06): -- Server was still alive, database was not available. -- Tried shutdown and restart. (failed) -- Tried Completed Recovery. (failed) -- Time based Incomplete Recovery successful. Course ID

35 Testing Process, continued. Experienced rebuild PPRD server from scratch (09/16/06): -- Multiple hard disks failure due to power surge. -- It was painful, took a long time. -- Reinstalled everything. (OS, ORACLE, COBOL, PERL etc) -- Database is recovered by using last night s cold backup. Set up physical standby database for TEST. Recovery from full image has not been tested yet. Course ID

36 SUNGARD SUMMIT 2007 sungardsummit.com 36 Topic #5 Implement A Completed DR plan A Community of Learning

37 Implement A Completed DR Plan Once it is determined that disaster conditions exist, the DR plan is to be implemented immediately. Make sure the college phone book on-hand is the most current version. Each DR team member knows his/her responsibility in a disaster scenario. Detailed plan of actions. -- Contain step-by-step instructions; -- Can be followed to do reinstall and recovery, -- Even by a person w/o training in a emergency situation. Course ID

38 Course ID

39 Course ID

40 Implement A Completed DR Plan, continued. Set up cloning server environments for testing purpose. Purchased necessary software and hardware for image. Perform image for mission critical severs. Scheduled cold backup and hot backup to run. Course ID

41 SUNGARD SUMMIT 2007 sungardsummit.com 41 Summary A Community of Learning

42 Summary Main issues Pre-disaster planning Develop your DR plan Specify disaster recovery team members. Specify the level of priorities in your recovery list. General Procedures preparing for a disaster. Software and Hardware preparing for a disaster. Recovery Strategies. Write down your plan of actions in details. DR plan testing Implement the Completed DR Plan Course ID

43 Acknowledgments I gratefully acknowledge the help and ideas from our DR team members: John Alderman, Network Analyst Mike Grzeszczak, System Administrator Mike Killian, VP Finance & Administrator Laurie Orzechowski, Director of Administrative System I gratefully acknowledge the ideas taken from the following disaster recovery plans: Disaster Recovery Journal, Fall 2006 * Volume 19 * Number 4 Disaster Planning & Emergency Federal Student Aid Conference, Bob Quinn, Penn State University, Pennsylvania ACU Technology Disaster Preparedness and Recovery Plan, Abilene Christian University, Texas SIGNATURE BANK Disaster Recovery Plan, SIGNATURE Bank, Ohio Course ID

44 Questions & Answers Questions? Course ID

45 Thank You! Jing Ding Please complete the online class evaluation form Course ID 767 SunGard, the SunGard logo, Banner, Campus Pipeline, Luminis, PowerCAMPUS, Matrix, and Plus are trademarks or registered trademarks of SunGard Data Systems Inc. or its subsidiaries in the t U.S. and other countries. Third-party names and marks referenced herein are trademarks or registered trademarks of their respective owners SunGard. All rights reserved. 45