Business Continuity Best Practices

Transcription

1 BusinessContinuityBestPractices MBAOperations&TechnologyConference April2008 Presenter: RajPatel,Partner Plante&MoranPLLC

2 BusinessContinuityBestPractices SessionObjectives KeyLearningConcepts: IntroductiontoDRP/BCP What swrongwithcurrentplans? ProcessforDevelopinganEffectivePlan RegulatoryAuthority&Guidance Howlongcanyourorganizationcopewiththelossofitskeyresources(People, Technology,Facilities,Suppliers&Customers)? Whatwouldbethebusinessimpactifcriticalinformationwasunavailabledueto disasterorsystemfailure? Doyouhaveproceduresinplaceformaintainingyourbusinessoperationsduringan unexpecteddisruption? Todownloadthispresentation,pleasegoto: 2

3 BusinessContinuityBestPractices Terminology DisasterRecoveryPlan Traditional1990sterminology Includedplansfordisastersandemergencies Moreeventfocusedthanprocessfocused OftentheITmanagerorVPOperations Responsibility Typicallytesting,wasdoneonlyatthe EDPHotsite BusinessContinuityPlan TerminologyofficiallyadoptedbyFFIEC Businesscontinuityplanningistheprocesswhereby financialinstitutionsensurethemaintenanceor recoveryofoperations,includingservicestocustomers, whenconfrontedwithadverseeventssuchasnatural disasters,technologicalfailures,humanerror,or terrorism FFIECInformationTechnologyExamination Handbook BusinessContinuityPlanning FFIECDefinition Contingency planning is the process of identifying critical information systemsandbusinessfunctions,and developing plans to enable those systems and functions to be resumedintheeventofadisruption. The process includes testing the recovery plans to ensure they are effective.duringthetestingprocess managementshouldalsoverifythat businessunitplans complement the informationsystemplans. 3

4 BusinessContinuityBestPractices Relevance BusinesscontinuityplanningisrequiredbytheregulatoryagenciesoftheFFIEC andguidelinesforplandevelopmentandmaintenanceareprovided intheffiec InformationTechnologyExaminationHandbook,BusinessContinuity Planning Businesscontinuityplanningisasoundbusinesspracticeinany organization regardlessofregulatoryrequirements Eventsofthepastsixyearshavesignificantlyincreasedtheneedforconcise attentiontoemergencypreparedness: Increaseddependencyondistributedtechnology,vendors,etc. Increasedbusinessdisasters(poweroutage,connectivityissues, InternetBankingsite down,etc.) Increasednumberofnaturaldisasters(Katrina,tornados,floods,etc.) Heightenednationalalertlevels terroristthreat 4

5 BusinessContinuityBestPractices What swrongwithcurrentplans? Outdatedorgatheringdustontheshelves Readslikeapolicyvs.aprocesstorestore Recoveryteamnotawareofplancontentsortrained OnlyaddressesrestoringITsystems Lacksaneffectiveplanto Restoreconnectivitybetweenlocations Managecommunicationstocustomers,localmedia,employees Neverbeentested Writtenlikea planfordummies Alargesingledocument Savedonlyonthenetwork Doesnotaddresssecurityincidents Toomuchfocusoncatastrophicdisastersornaturaldisasters Doesnotaddressavailabilityofcriticalvendors Oneplanfitsalldisruptions 5

6 BusinessContinuityBestPractices MaturityModel FUNDAMENTAL Responsibilitiesforcontinuousserviceareinformal,withlimitedauthority.Managementisbecomingawareoftherisksrelatedto andtheneedforcontinuousservice.thefocusisontheitfunction,ratherthanonthebusinessfunction.usersareimplementing workarounds.theresponsetomajordisruptionsisreactiveandlargelyunprepared.plannedoutagesarescheduledtomeetit needs,ratherthantoaccommodatebusinessrequirements. TRANSITIONAL Responsibilityforcontinuousserviceisassigned.Theapproachestocontinuousservicearefragmented.Reportingonsystem availabilityisincompleteanddoesnottakebusinessimpactintoaccount.therearenodocumenteduserorcontinuityplans, althoughthereiscommitmenttocontinuousserviceavailability, anditsmajorprinciplesareknown.areasonablyreliableinventoryof criticalsystemsandcomponentsexists.standardizationofcontinuousservicepracticesandmonitoringoftheprocessisemerging, butsuccessreliesonindividuals. ENHANCED INTEGRATED Accountabilityisunambiguousandresponsibilitiesforcontinuousserviceplanningandtestingareclearlydefinedandassigned. Plans aredocumentedandbasedonsystemcriticalityandbusinessimpact.thereisperiodicreportingofcontinuousservicetesting. Individualstaketheinitiativeforfollowingstandardsandreceivingtraining.Managementcommunicatesconsistentlytheneedfor continuousservice.highavailabilitycomponentsandsystemredundancyarebeingappliedpiecemeal.aninventoryofcritical systemsandcomponentsisrigorouslymaintained. Responsibilitiesandstandardsforcontinuousserviceareenforced.Responsibilityformaintainingthecontinuousserviceplanis assigned.maintenanceactivitiestakeintoaccountthechangingbusinessenvironment,theresultsofcontinuousservicetestingand bestinternalpractices.structureddataaboutcontinuousserviceisbeinggathered,analyzed,reportedandactedupon.trainingis providedforcontinuousserviceprocesses.systemredundancypractices,includinguseofhighavailabilitycomponents,arebeing consistentlydeployed.redundancypracticesandcontinuousserviceplanninginfluenceeachother.discontinuityincidentsare classifiedandtheincreasingescalationpathforeachiswellknowntoallinvolved. OPTIMIZED Integratedcontinuousserviceprocessesareproactive,selfadjusting,automatedandselfanalyticalandtakeintoaccount benchmarkingandbestexternalpractices.continuousserviceplansandbusinesscontinuityplansareintegrated,alignedand routinelymaintained.buyinforcontinuousserviceneedsissecuredfromvendorsandmajorsuppliers.bankwidetestingoccursand testresultsarefedbackaspartofthemaintenanceprocess.continuousservicecosteffectivenessisoptimizedthroughinnovation andintegration.gatheringandanalysisofdataisusedtoidentifyopportunitiesforimprovement.redundancypracticesand continuousserviceplanningarefullyaligned.managementdoesnotallowsinglepointsoffailureandprovidessupportfortheir remedies.escalationpracticesareunderstoodandthoroughlyenforced. 6

7 BusinessContinuityBestPractices TheBasics 7

8 BusinessContinuityBestPractices TheProcess 8

9 BusinessContinuityBestPractices 1.Mission,Objectives,Scope&Assumptions Mission&Objective Examples Themostimportantobjectiveofbusinesscontinuityplanningistoprotectthe Bank ifallorpartsofits operationsorcomputerservicesaredisruptedbyadisaster.the planningprocessshouldreducetoa minimum,thedisruptionofoperationsandensuresomeleveloforganizationalstabilityduringanorderly recoveryafteradisaster. Otherpossibleobjectivesare:Managesuccessfullythroughadisaster,meetregulatoryandcontractual requirements,ensurecontinuationofbranchoperations. Assumptions Examples Theplanisdesignedtorecoverfromthe"worstcase"destructionofthe Bank operatingenvironment.the worstcaseincludesanynondataprocessingfunctionthatmaybeincloseproximitytothedatacenteror workstations. Thisplanisnotdesignedforthe worstcase destruction,butfocusesonthelossofrecoveryofkey componentssuchaslocalapplication,network,etc. Fiserve isresponsiblefortheavailabilityofcoreapplications(suchasiti)andthusnotaddressedinthisplan. Theplanisbaseuponasufficientnumberofstaffnotbeingincapacitatedtoimplementandaffectrecovery. Therefore,thelevelofdetailoftheplaniswrittentoastaff experiencedinthe Bank s computerservices. Development,testingandimplementationofnewtechnologiesandapplicationsaresuspendedsothatall resourcesareavailabletorecoverexistingcriticalproductionprocessing. Analternatesite(backupcomputerfacility)inwhichtoestablishrecoveryofcomputerprocessingmaybe necessary.timeframerequirementstorecovercomputerprocessingaresignificantlylessthanestimated timestorepair/reconstructadatacenteronanemergencybasis. Thecomputerfacilitiesofthealternativesiteisnotwithinthescopeofthisplanandisassumednottobe impactedbyanydisasterwhichmayinterruptcomputeroperations at Bank offices. 9

10 BusinessContinuityBestPractices 2.PlanCoordinator&DevelopmentTeam CharacteristicsofBCPCoordinator: Shouldhaveauthority Shouldhaveavailabletime/resources Shouldbeabletocommunicatewithtechnicalstaffandnontechnicalstaff Shouldbeorganized,detailorientedandacompetentwriter Fluentinprojectmanagementprinciplesandtechniques Willneedhighlydevelopedqualitiesofpatience,perseveranceanddiplomacy Coordinatorsneedtocultivateenthusiasmandconstantlyreinforcethebuyinofplanparticipants ThemakeupofyourteamwillvarydependingonthesizeofyouITorganization,business unitandthenumberofdepartmentsinvolved Determineactiveteammembersandadvisoryteammembers fromfunctionalareas suchas: Security(data&physical) SeniorManagement BranchOperations CustomerService HumanResources RiskManagement IT Lending Trust Facilities etc. 10

11 BusinessContinuityBestPractices 3.ProjectPlan FormalprojectplantodevelopyourBCPplan Treatitlikeyouwouldanyotherprojectwithformalplan,team,responsibilities,timelines,budget,etc. Don tletitbeapassiveproject Assignastrongprojectmanager Developkeytimelinesandmilestones Involveateamthatrepresentsyourorganization Sponsorshipattheexecutiveleveliscritical Allocateappropriateresources Don tletsoftwaredrivetheproject Hireoutsidehelp(projectmanagers,consultants,etc.) Facilitatetheprocess&notwritetheplanforyou Strongtools&methodologies Experiencewithfinancialinstitutions Bewaryofconsultantsthatpushtowardsaproductorrecoverysite(dowhat srightforyou) Bringstrongprojectmanagementskills(willkeeptheprojecton course) 11

12 BusinessContinuityBestPractices 3.ProjectPlan(cont.) PriortodevelopingPlan ReviewexistingDRPplan Reviewinternalplansandpolicies: Evacuationplans Fireprotectionplan Safety&healthprogram Securityprocedures Insuranceprogram Riskmanagementplans Meetwithoutsidegroups Askaboutpotentialemergenciesandavailableresourcesforresponding tothem.forexample,onefacilitydiscoveredthatadam50milesawayposedathreat.familiarizethe localemergencyagencieswithyourfacilityandanyspecificneeds Localemergencymanagementoffice Firedepartment Policedepartment Emergencymedicalservices Utilitycompanies communicationlines,water,electric,etc. OtherlocalorganizationsthatcouldpresentpotentialthreatstoBank Identifyyourinternalresources&capabilities: Personnel facilitiesmanager,electrician,networkadministrator,etc. Equipment fireprotection,communications,emergencypower,etc. Facilities emergencyoperatingcenter,etc. Backupprocesses arrangementswithotherfacilitiesororganizationstoprovideforcriticaloperationssuchaspayroll, communications,etc. Reviewinsurancepolicyforadequatecoverageforinfrastructure andrecoverycosts Conductdatacenter/facilityassessment 12

13 BusinessContinuityBestPractices 13 4.ThreatAssessment Riskevaluationinvolvesdeterminingtheeventsthatcanadverselyaffectthe Bank soperations,thedamagesucheventscancauseandthemeasuresneeded topreventorminimizetheeffectsofpotentialloss Riskevaluationwouldinclude: Threatidentification Determineprobability/occurrence Determineseverity/impact Identifypreventivemeasuresinplace Identifypreventivemeasuresimprovementopportunities CommonNaturalDisasters Earthquakes Hurricanes Floods/MudSlides Tornados Lightning ExtremeWeather CommonBusinessDisasters Communications/NetworkFailure HardwareFailure PowerFailure SoftwareFailureorCorruption VirusorHackAttack ChemicalSpills Fire HumanErrors ArmedRobbery Terrorism

14 BusinessContinuityBestPractices 4.ThreatAssessment(cont.) Thefollowingchartpresentsthetypesofeventswhichhaveforcedcompaniestodeclarea disaster: 14

15 BusinessContinuityBestPractices 5.ImpactAnalysis Recoveryproceduresarestagedaroundthemostcriticalresource (withthe shortestmad)totheapplicationwiththelongestmad Department Name: Operations Interviewee: John Doe Date: 7/26/2007 MN MD CT Minimal Impact Moderate Impact Critical Impact Application Name # of users Usability Daily/ Weekly/ Monthly/ Qtrly/ Annually < 8 Hours Essential 8-24 Hours Hours 2-3 Days 4-7 Days ChexSystems ~900 Daily MD MD MD MD MD MD MD CT Suspended 60 days pull credit report or alter process for validation Deluxe Checks ~900 Daily MN MN MN MN MN MN MN MN MN Suspended n/a mail check orders Digital Insight ~3500 Daily MN MD CT Essential 48hrs Use phone banking or come to branch MCIF 1 Daily MN MN MN MN MN MN MN MN MN Suspended N/A use other report writers Bill Payment 3,000 Daily MN MN MN MD CT Delayed 4 days customers can pay their bills other ways Core Banking System ~200 Daily MD CT Essential 12hrs???? Trust Metavante Watchdog OFAC WirePro Delayed 8-15 Days Days Suspended Days > 60 Days Recovery Strategy (Essential, Delayed, Suspended) Maximum Allowable Downtime What would you do if the system was not available? 15

16 BusinessContinuityBestPractices 6.StrategyAnalysis CURRENT STRATEGY Buy-and-Build Cold Site Warm Site Hot Site Hot-Mirrored Site Recovery Strategy Identify an alternate site, buy or lease equipment, re-build servers Designate a fully operational data center as alternate site in advance of disaster. Recovery similar to Buy-and-Build at designated site Establish alternate site with stand-by hardware and operating systems. Load applications and restore data from tape after a disaster Establish alternate site with stand-by hardware, operating system, and applications. Load data on a daily basis from tape Operate two remote data centers both for production processing. Traffic is dynamically routed between sites Recovery Time 5 days or more More than two days, exact time depends upon hardware availability 24 to 36 hours 3 to 12 hours Instantaneous Technical Architecture None Data center with environmental controls and telecommunications Load applications and restore data from tape during a disaster Restore data from tape on a daily basis before a disaster Mirroring Load Balancing Key Benefits Inexpensive Accommodates webbased systems Inexpensive Can use as alternate site for development and lab Good compromise between recovery time and cost Reliable recovery method Rapid recovery of critical applications Instantaneous recovery Risk of data loss limited to last few uncommitted transactions Operational efficiencies Key Weaknesses Potentially unreliable Can not accommodate web-based systems May take up to a week to recover Potentially unreliable May take up to a week to recover Loss of data since most recent back-up If servers are used for development recovery may be hindered by configuration changes Loss of data since most recent back-up Can t use alternate site for test or lab purposes Expensive Loss of data since most recent back-up Expensive Potentially complex to operate 16

17 BusinessContinuityBestPractices 6.StrategyAnalysis Selectingyouroffsitevendor: Reputation Howlonghasthefacilitybeeninexistence?Haveyoucheckedthier financialstatements? SiteSecurity Securityatstoragefacilityshouldbenolessstringentthanyourownfacility.Somequestionsto consider: Whataretheaccesscontrolsinthefacility? Isvisitoraccessrestricted? Areclientnamesconcealed,evenfromoneanother? Securitymeasuresduringtransportationofmedia?(unmarkedcars,securityinvehicles,employeemonitoring,etc.?) Howareemployeesscreened? Arecamerasorotherdevicesusedtomonitorfacilitytraffic? Howareemergencycallshandled? MediaManagement Averyimportantfactor,howthefacilitymanagesclientrecords,maypresentthefollowing questions: Howismediaofseveralclientssegregated? Ismediatransportedinplasticcontainersorcardboardboxes? Whatkindofinventorymanagementsystemisused? Areemployeestrainedinpropermediahandling? Whatcontrolsexisttomonitorflowofmediainandoutofthefacility? EnvironmentalFactors detection,preventionandsuppressioncontrolsforsmoke,fire, water,humidity,etc. Whatcapabilitiesareinstalledtodetectsmoke,heat,flame,water,andintrusion? Whatsuppressionsystemsexist? Isthealarmsystemtieddirectlytofire,police,andsecurityservices? Howaretemperature,heat,humidityandcontaminationcontrolled? Howoftenareenvironmentalcontrolstested? Transportation Tapesandrecordsareathigherriskoflossordamagewhileridinginthebackofavan Ismediatransportedbyvendoremployeesorindependentcontractors? Ismediasubjectedtoambientclimateconditions? Arevehiclesequiped withantitheftdevices? 17

18 BusinessContinuityBestPractices 7.DocumentPlan Finallytimetodocumentplan: ItisimperativetocommittheBusinessContinuityPlantowriting,otherwise: Planningwillbeforgottenwhenanincidentoccurs Therewillbenoconsistencytoactionsandresponsestaken Therewillnotbeabaselinetoupdateandimproveovertimeand aschangesoccur Manysitesaredependentuponothersitesforproductionandnetworkfunction thereneedstobeacommon understandingofwhatwillbedoneatthevariousbanksitestopreservebusinesscontinuity Regulatorycompliance PlanStructure Logicallysegregatedsections o Administrativesections:Team,roster,responsibilities,whento declareadisaster,incidentresponsepolicy,etc. o Policies:Incidentresponsepolicy,planmaintenancepolicy,plantestingpolicy,etc. o AssetInventory:Applicationlisting,vendorinformation,networkdiagrams,etc. o Analysis:ThreatAssessment,BusinessImpactAnalysis,etc. o RecoverySteps:Systemsrecovery/restoration,etc. o ContinuityofOperations:Branchoperations,lending,etc. o Attachments:Damageassessment,samplepressrelease,phoneredirectphonegreeting,etc. Planscanbesegregatedby: Durationofdisruption:24hrs,72hrs,5days Typeofdisruption:Systems,neighborhood,branchlevel,etc. Typeofdisaster:Fire,communicationbreak,etc. 18

19 BusinessContinuityBestPractices 8.Implementation/Maintenance Security Becauseofthesensitivenatureoftheinformationyourplanwillcontain,it ssuggestedthatonlythosepersonswhohave beendesignatedasmembersoftherecoveryteamshouldbegivencopiesofyourplan Planstorage Singledocumentvs.fragmented Physicalcopyvs.electroniccopy OnCDsvs.Internet Copiesoftheplanshouldbeeasilyaccessible Severalcopiesoftheplanshouldbestoredoffsiteinasecurelocation Ifplanisdistributedonline,makesurethathostingisseparatefromtheproductionenvironment Keyemployeesmayneedaccesstotheplanduringnonworkinghours Eachplanshouldbekeptcurrent,datedandversioncontrolled Ifsoftwareprogramhasbeenusedtoassistwithplandevelopment,copiesoftheplanningdisksandprogramshouldbe storedoffsite Maintainingtheplanisasimportantaswritingtheplanitself Mostrecoveryplansarenotmaintained.Withinayearorless,theplanbecomesoutdated,asstaffhavechanged,the infrastructurehaschanged,thevendorshavechanged,etc. Presentpartsoftheplantothosewhoassistedincreatingtheplaninitiallyforupdates Instituteamaintenanceplanthatincludesautomaticreminderswhereeachsectionisdesignatedwiththefrequency (quarterly,annually,etc.) Theboardisrequiredtoreviewandapprovetheplanannually 19

20 BusinessContinuityBestPractices 9.TestingthePlan Checklisttesting(alsoknownaswalkthru) Determineswhethertheplanisadequate,i.e.,therecoveryteam reviewstheplanandidentifieskeyelementsthatshouldbe uptodateandavailable,thetelephonenumberlistingsarecurrent,copiesofplanarestoredatalltherightlocations,the inventoryofsystemsisaccurate,theriskassessmentiscurrent,etc. Advantages:Itischeap,involvesminimalinterruptiontobusiness,canusuallybearrangedwithshortnotice,itisagentleway toexploreandtesttheplan Disadvantage:Ithaslimitedtrainingvalue,thetestlacksrealism Nonbusinessinterruptiontest(alsoknownasroleplay) Adisasterissimulatedsothatnormaloperationsarenotinterrupted.Thefollowingareasareadequatelytested:hardware, software,telecommunications,supplies,etc. Advantages:Goodtrainingvalue,challengestheparticipantsand plan Disadvantages:Theroleplaycanmovetowardsextreme ParallelTesting Underthisscenariothesystems(withprecedingday sbackupdata)arerestoredatalternatesiteandcurrentdays transactionsprocessed Allreportsnormallyproducedatthealternatelocationforthecurrentdayshouldagreewiththosereportsatyournormal businesslocation Advantages:excellenttrainingvalue,teststherecoveryofkeysystems Disadvantages:Itwillbecostly BusinessInterruptionTesting(pulltheplug) Thisteststhetotalbusinesscontinuityplan Thistestiscostlyandcoulddisruptyournormalbusinessoperations,soproceedwithcaution Adequatetimemustbeallocatedforthistest Youmaywanttotestonlycertainportionsoftheplaninitially toidentifytheworkabilityofeachpartpriortoattemptingthe fulltest 20

21 BusinessContinuityBestPractices RegulatoryAuthority&Guidance FFIECBusinessContinuityPlanning(BCP)Booklet FDICFinancialInstitutionLetter FIL (newguidanceonbcp&supervisionoftechnologyserviceproviders) FDICFinancialInstitutionLetter FIL (BoardofDirectors&SeniorManagementresponsibilityoverBC) OCCBulletin (BCPBooklet) OCCBulletin200314(LargeNationalBanks) (stepstoprotectu.s.financialsystems) FDICFinancialInstitutionLetterFIL (InfluenzaPandemicPreparedness) 21

22 BusinessContinuityBestPractices RegulatoryAuthority&Guidance GLBA501(b) SecurityGuidelines Asstatedinsection501,thesesafeguardsareto: (1) Insurethesecurityandconfidentialityofcustomerrecordsandinformation; (2) Protectagainstanyanticipatedthreatsorhazardstothesecurityorintegrityof suchrecords;and (3) Protectagainstunauthorizedaccessto,oruseof,suchrecordsorinformationthat wouldresultinsubstantialharmorinconveniencetoanycustomer TheseGuidelinesaddressstandardsfordevelopingandimplementingadministrative, technical,andphysicalsafeguardstoprotectthesecurity,confidentiality,andintegrity ofcustomerinformation Therefore: SecurityStandardsdonotspecificallystatethattheBank sinformation SecurityProgramneedaddressthreatstoinformationavailability However, management sriskassessment(s)shouldconsiderthreatstosecurity, IntegrityandAvailability 22

23 BusinessContinuityBestPractices BeInformed 23

24 BusinessContinuityBestPractices BeInformed Hazards Assessment is intended to provide emergency managers, planners,forecastersandthepublic advancenoticeofpotentialhazards related to climate, weather and hydrological events. It integrates existing National Weather Service official medium (35 day), extended (610 day) and longrange (monthly and seasonal) forecasts and outlooks, and hydrological analyses and forecasts, which use stateof theart science and technology in theirformulation. 24

25 BusinessContinuityBestPractices ThankYou Todownloadthispresentation,pleasegoto: Presenter: RajPatel,Partner Plante&MoranPLLC