HathiTrust is a Solution

Transcription

1 HathiTrust isa Slutin TheFundatinsfa DisasterRecvery PlanfrtheShared DigitalRepsitry Thisreprtservesas recmmendatinsmadeby MichaelJ.Shallcrss, 2009DigitalPreservatinIntern UniversityfMichigan SchlfInfrmatin

2 ii ExecutiveSummary ThisreprtseekstestablishtheframewrkfaDisasterRecveryPlanfrtheHathiTrust DigitalLibrary.Whileprfessinalbestpracticesandinstitutinalneedshaveprvidedaclearmandate frhathitrust sdisasterrecveryprgram,cmmnparlancehasftenbscuredtwprminent featuresfsuchinitiatives.first,a DisasterRecveryPlan isactuallycmprisedfasuitefdcuments whichdetailarangefissues,frmcrisiscmmunicatinsandthecntinuityfadministrativeactivities ttherestratinfhardwareanddata.secnd,thereisncnclusinttheplanningprcess;itis insteadacntinuuscyclefbservatin,analysis,slutindesign,implementatin,training,testing, andmaintenance. Theprimarygalfthepresentdcumentistprvideafundatinnwhichfutureplanning effrtsmaybuild.tthatend,itexaminesthestrategiesbywhichhathitrusthasanticipatedand mitigatedtheriskspsedbytencmmnscenariswhichculdprecipitateadisaster: Hardwarefailureanddatalss Netwrkcnfiguratinerrrs Externalattacks Frmatbslescence Creutilityrbuildingfailure Sftwarefailure Operatrerrr Physicalsecuritybreach Mediadegradatin Manmadeaswellasnaturaldisasters. Asthislistreveals,adisasterwithinthedigitalrepsitryrefersntmerelytdatalss,thedestructin fequipment,rdamagetitsenvirnment,buttanyeventwhichhastheptentialtcausean extendedserviceutage.freachscenari,thereprtdiscussespssiblethreats,summarizesthe ptentialseverityfrelatedevents,andthendetailsslutinshathitrusthasenactedthrughdirect qutatinsfrmthehathitrustwebsiteandtracself assessment,servicelevelagreements,and literaturefrmserviceprvidersandvendrs.attachedappendicesprviderelevantinfrmatinand includecntactsfrimprtanthathitrustresurces,ananntatedguidetdisasterrecveryplanning references,andanverviewfkeystepsinthedisasterrecveryplanningprcess. ThecncludingsectinfthereprtprvidesrecmmendatinsandactinitemsfrHathiTrust asitprceedswithitsdisasterrecveryinitiative.thesearedividedintshrt(0 6ms.),Intermediate (6 12ms.)andLng Term(12+ms.)bjectivesandarearrangedinasuggestedrderf accmplishment. Shrt termgalsinclude: DescribingthenatureandextentfHathiTrust sinsurancecverage Testingandvalidatinfcurrenttapebackupprcedures Imprvedphysicalandintellectualcntrlversystemhardware Establishment,distributin,andmaintenancefphnetrees Increaseddcumentatinfinstitutinalknwledge IdentificatinfDisasterRecverymeasuresinplaceattheIndianaplissite. Intermediate termbjectivesfcusn: CreatinfaDisasterRecveryPlanningCmmittee

3 Initiatinfthedatacllectinandanalysisessentialtthecreatinfrecvery strategies(thissectinprvidesahighlevelbreakdwnfvariustasksand includesthecrdinatinfactivitiesbetweentheannarbrandindianaplis sitesaswellaswithserviceprvidersandvendrs.) Lng termactinitemsdealwith: CmpletinandimplementatinfthesuitefDisasterRecverydcuments Initiatinfstafftrainingandtestsfrganizatinalcmpliance. Stragefanadditinalcpyfbackuptapesataremtethirdlcatin InvestigatinfanalternatehtsiteinAnnArbrintheeventadisaster rendersthemaccunusable Cnsideratinfathirdinstanceftherepsitry Avidancefvendrlck inifakeysuppliershuldgutfbusiness. Thisreprtdemnstratesthatvariusriskmanagementstrategies,designelements,perating prcedures,andsupprtcntractshaveendwedhathitrustwiththeabilitytpreserveitsdigital cntentandcntinueessentialrepsitryfunctinsintheeventfadisaster.theestablishmentfthe Indianaplismirrrsite,theperfrmancefnightlytapebackupstaremtelcatin,andthe redundantpwerandenvirnmentalsystemsfthemaccreflectprfessinalbestpracticesandwill enablehathitrusttweatherawiderangeffreseeableevents.unfrtunately,disastersftenresult frmtheunknwnandtheunexpected;whiletheafrementinedstrategiesarecrucialcmpnentsf adisasterrecveryplan,theymustbesupplementedwithadditinalpliciesandprcedurestensure that,cmewhatmay,hathitrustwillbeabletcarrynasbthanrganizatinandadedicatedservice prvider. iii

4 Acknwledgements TheauthrwuldliketthankShannnZacharyfrherencuragementandguidance;Cry SnavelyandJeremyYrkfrtheirgenerusexpenditureftime,energy,andknwledge;andNancy McGvernandLanceStuchellfraccessttheirutstandingDisasterRecveryPlanningresurces.The fllwingindividualshavealsbeeninvaluablesurcesfadvice,supprt,andinfrmatin:jhnwilkin, BbCampe,CyndiMesa,AnnThmas,JhnWeise,LarryWentzel,LaraUnger Syrigs,BillHall,Emily Campbell,SebastienKrner,JessicaFeeman,PhilFarber,ChrisPwell,CamernHanver,Stephen Hipkiss,TimPrettyman,ReneGbeyn,andKrystalHall.ThanksalstDr.ElizabethYakel,MagiaKrause, andvernicaandcrafambrugh.thewrkinthisreprtwasmadepssiblebyanimlsgrant. iv

5 TablefCntents ExecutiveSummary p.ii Acknwledgements p.iv Intrductin p.1 GalsfrHathiTrust sdisasterrecveryprgram p.1 TheMandatefrDisasterRecveryPlanninginDigitalPreservatin p.2 DisasterPreparednessintheDesignandOperatinfHathiTrust p.2 EssentialHathiTrustBusinessFunctins p.3 HathiTrust sdisasterrecverystrategies p.5 BasicRequirementsfrDisasterRecvery p.5 DisasterRecveryStrategy#1:RedundancybetweentheAnnArbrandIndianaplisSitesp.5 DisasterRecveryStrategy#2:NightlyAutmatedTapeBackups p.6 Scenari1:HardwareFailurerObslescenceandDataLss p.8 Review:RisksInvlvingHardwareFailurerObslescenceandDataLss p.8 HathiTrust sslutinsfrhardwarefailureanddatalss p.8 RedundantCmpnentsandSinglePintsfFailureintheHathiTrustInfrastructure p.9 KeyFeaturesfHathiTrust sisilniqclusteredstrage p.10 HardwareSupprtandService p.12 EquipmentTracking p.13 HardwareReplacementSchedule p.13 TimelinefrEmergencyReplacementfHathiTrustInfrastructure p.13 HathiTrustandInsuranceCverageattheUniversityfMichigan p.14 Scenari2:NetwrkCnfiguratinErrrs p.15 Review:RisksInvlvingNetwrkCnfiguratinErrrs p.15 HathiTrust sslutinsfrnetwrkcnfiguratinerrrs p.15 ExtentfITCmSupprt p.15 ITCmRespnsibilities p.16 ITCmServicesinRespnsetOutagesrDegradatinImpactingtheNetwrk p.16 HathiTrustRespnsibilities p.16 Scenari3:NetwrkSecurityandExternalAttacks p.17 Review:RisksInvlvingNetwrkSecurityandExternalAttacks p.17 HathiTrust sslutinsfrnetwrksecurity p.17 Scenari4:FrmatObslescence p.18 Review:RisksInvlvingFrmatObslescence p.18 HathiTrust sslutinsfrfrmatobslescence p.18 SelectinfFileFrmats p.18 FrmatMigratinPliciesandActivities p.19 Scenari5:CreUtilityand/rBuildingFailure p.20 Review:RisksInvlvingCreUtilityrBuildingFailure p.20 HathiTrust sslutinsfrutilityrbuildingfailure p.20 GeneralMaintenanceandRepairsinUniversityfMichiganFacilities p.20 TheMichiganAcademicCmputingCenter(MACC) p.20 ArbrLakesDataFacility(ALDF) p.22 v

6 Scenari6:SftwareFailurerObslescence p.23 Review:RisksInvlvingSftwareFailurerObslescence p.23 HathiTrust sslutinsfrsftwareissues p.23 Scenari7:OperatrErrr p.24 Review:RisksInvlvingOperatrErrr p.24 HathiTrust sslutinsfroperatrerrr p.24 Ingest p.24 ArchivalStrage p.24 Disseminatin p.24 DataManagement p.24 Scenari8:PhysicalSecurityBreach p.25 Review:RisksInvlvingaPhysicalSecurityBreach p.25 HathiTrust sslutinsfrphysicalsecurity p.25 SecurityattheMACC p.25 SecurityattheALDF p.26 Scenari9:NaturalrManmadeDisaster p.27 Review:RisksInvlvingaNaturalrManmadeDisaster p.27 HathiTrust sslutinsfrnaturalrmanmadecatastrphicevents p.27 BasicDisasterRecveryStrategies p.28 Scenari10:MediaFailurerObslescence p.29 Review:RisksInvlvingMediaFailurerObslescence p.29 HathiTrust sslutinsfrmediafailure p.29 RemainingVulnerabilities p.29 CnclusinsandActinItems p.30 Cnclusins p.30 Shrt TermActinItems p.30 Intermediate TermActinItems p.31 Lng TermActinItems p.32 APPENDIXA:CntactInfrmatinfrImprtantHathiTrustResurces p.34 APPENDIXB:HathiTrustOutagesfrmMarch2008thrughApril2009 p.37 APPENDIXC:WashtenawCuntyHazardRankingList p.38 APPENDIXD:AnntatedGuidetDisasterRecveryPlanningReferences p.39 APPENDIXE:OverviewftheDisasterRecveryPlanningPrcess p.45 APPENDIXF:TSMBackupServiceStandardServiceLevelAgreement(2008) p.52 APPENDIXG:ITCS/ITCmCustmerNetwrkInfrastructureMaintenanceStandardService Agreement(2006) p.53 APPENDIXH:MACCServerHstingServiceLevelAgreement(Draft,2009) p.54 APPENDIXI:MichiganAcademicCmputingCenterOperatingAgreement(2006) p.55 **AppendicesF IareembeddedPDFfiles.** vi

7 Intrductin Intherealmfprintlibraries,adisasterisafairlyunambiguusevent:itisafire,abrkenpipe, aninfestatinfpests inshrt,anythingwhichthreatensthecntinueduseandexistenceftextsr theenvirnmentinwhichtheyarestred.thisbasicdefinitinmayalsbeappliedtthedigitallibrary, inwhichadisasterrefersntmerelytthelssfcntentrcrruptinfdata,thedestructinf equipmentrdamagetitsenvirnment,buttanyeventwhichhastheptentialtcausean extendedserviceutage.thislastpartprvestbethegreatestdifferencebetweentheprintand digitalwrldsbecausethereareagreatmanythreatswhichcanleavedataintactbutincapacitatethe primaryfunctinsfadigitallibrary.thedailyperatinfaninstitutinsuchashathitrustinvlvesthe anticipatinandreslutinfavarietyfprblems crashedservers,sftwarebugs,netwrkingerrrs, etc. whichnlyrisetthelevelfa disaster whentheyexceedthecapacityfnrmalperating prceduresand/rthemaximumallwableutageperids.disasterrecveryplanningthusprmptsus tdevelprbuststrategiestmitigateandlimittheeffectsfcmmnprblemsandatthesametime frcesustthinktheunthinkable.nevertheless,cnfrntingwrst casescenarisisavitalactivity;the beliefthataneventwillneverhappensimplybecauseithasneverhappenedisaninvitatintthevery disasterweseektavid.hereinliesacnundrum,inthatthecreatinfdetailedplansfrevery eventualityisnearlyimpssibleandalsimpractical,sincetheresultsfsuchanendeavrwuldbe needlesslycmplexaswellasexpensive.atitsbasis,then,disasterrecveryplanningdemandsan astuteassessmentfrisksthatwemayweighthecstsfpreparatinsandslutinsagainstthecsts faptentialevent. Swheretbegin?WhenthesubjectfDisasterRecveryPlanningarises,cmmnparlance ftenbscurestwprminentfeaturesfsuchinitiatives.first,a DisasterRecveryPlan isactually cmprisedfasuitefdcumentswhichdetailavarietyfrelatedissues,frmcrisiscmmunicatins andthecntinuityfadministrativeactivitiesttherecveryfhardwareanddataandtherestratin fcrefunctins.secnd,thereisncnclusinttheplanningprcessrapintatwhichaplanis dne ;thereisinsteadacntinuuscyclefbservatin,analysis,slutindesign,implementatin, training,testing,andmaintenance.theessentialfirststepistherefreathrughknwledgefthe rganizatin,itsgals,anditsmandatefradisasterrecveryprgramsthatlatereffrtscanfcus nthearticulatinfpliciesandthedevelpmentfslutins.asapreliminarystepinthiseffrt,this reprtlkstestablishabasicfundatinfrmwhichfutureplanningeffrtsmaygrw. GalsfrHathiTrust sdisasterrecveryprgram WhileamrefrmalstatementfHathiTrust sgalsandrequirementsfritsdisasterrecvery Prgrammustbeelucidated,therepsitry smissinstatementprvidesagdindicatinfitsmain bjectiveinthefrmatinfadisasterrecveryplan.aspartfitsaimt cntributetthecmmn gdbycllecting,rganizing,preserving,cmmunicating,andsharingtherecrdfhuman knwledge, HathiTrustseeks thelppreservetheseimprtanthumanrecrdsbycreatingreliableand accessibleelectrnicrepresentatins. 1 Thisstatementclearlyjinsthetwinimperativesfpreservatin andaccesswithanadditinalrequirement:reliability.thedevelpmentandimplementatinfa DisasterRecveryPlanwillensurethatdigitalbjectswillretaintheirauthenticityandintegrityverthe lngtermandthatpartnerlibrariesanddesignatedusersmayrelynhathitrustservices(rtheirtimely resumptin)andcntentinthefacefcatastrphicevents. 1 HathiTrust. Missin&Gals (2009)retrievedfrmhttp://

8 TheMandatefrDisasterRecveryPlanninginDigitalPreservatin HathiTrust smandatefracmprehensiveandpractivedisasterrecveryplanstemsfrma numberfsignificantsurces,amngwhichwemayincludeitsmissinandgals.the Institutinal DataResurceManagementPlicy (2008)ftheUniversityfMichigan sstandardpracticeguideals prvidesanimpetusfrthecreatinfadisasterrecveryprgram.whilentnecessarilyinclusivef themichigandigitizatinprjectmaterialsstredinhathitrust,thisdcumentunderscreshw imprtantitisthatdataresurces besafeguarded[and]prtected and cntingencyplans[ ]be develpedandimplemented. 2 Initsdiscussinfthelatterpint,theplicyspecifiesthat: DisasterRecvery/BusinessCntinuityplansandthermethdsfrespndingtanemergency rtherccurrencesfdamagetsystemscntaininginstitutinaldata[ ]willbedevelped, implemented,andmaintained.thesecntingencyplansshallinclude,butarentlimitedt, databackup,disasterrecvery,andemergencymdeperatinsprcedures.theseplanswill alsaddresstestingfandrevisintdisasterrecvery/businesscntinuityprceduresanda criticalityanalysis. 3 Whiledatabackupprceduresandahstfriskmanagementpracticesarealreadyanintegralpartf HathiTrust speratin,therepsitrynwlkstfrmalizethetherstrategiessuggestedbythe InstitutinalDataManagementPlicy. Beyndtheexamplelaidutbythisdcument,HathiTrust s mandatefrdisasterrecveryderivesfrmtheprfessinalliteraturedetailingbestpracticesinthe fieldfdigitalpreservatin.thereferencemdelfranopenarchivalreferencesystemidentifies DisasterRecveryasanessentialcmpnentfits ArchivalStrage functinandhighlightsthe imprtancefsuchplansinachievingthegalflng termpreservatinfadigitalarchive shlding.as utlinedintheoaisdcument, thedisasterrecveryfunctinprvidesamechanismfrduplicating thedigitalcntentsfthearchivecllectinandstringtheduplicateinaphysicallyseparatefacility. 4 HathiTrusthassuccessfullymetthisrequirementbyperfrmingnightlytapebackupsandestablishinga mirrrsiteatindianauniversityinindianaplis.thetrustedrepsitriesaudit&checklist:criteriaand Checklist(2007)isevenmreexplicitinitsrequirementthatrepsitriesdcumenttheirpliciesand prcedureswith suitablewrittendisasterpreparednessandrecveryplan(s),includingatleastneffsitebackupfallpreservedinfrmatintgetherwithanff sitecpyftherecveryplan(s). 5 Prfessinalbestpracticesaswellasinternalneedsandgalsthusprvidethemandatewhichunderlies HathiTrust sdevelpmentfafrmaldisasterrecveryplan. DisasterPreparednessintheDesignandOperatinfHathiTrust OneftheprimarygalsfHathiTrustistprvide transparencyinallfitsperatins, includingitswrktcmplywithdigitalpreservatinstandardsandreviewprcesses. 6 Nwhereisthis cmmitmentmreclearthaninitseffrtstanticipateandmitigateriskswhichculdthreatenthe 2 UniversityfMichigan. InstitutinalDataResurceManagementPlicy (2008)StandardPracticeGuide, retrievedfrmhttp://spg.umich.edu/n8july Ibid. 4 CnsultativeCmmitteefrSpaceDataSystems.ReferenceMdelfranOpenArchivalInfrmatinSystem (2002)p OCLCandCRL. SectinC3.4 TrustedRepsitriesAudit&Checklist:CriteriaandChecklist(2007)p HathiTrust. Accuntability (2009)retrievedfrmhttp://

9 cntentsandfunctinsftheshareddigitalrepsitry.asafirststepinaddressingthedisaster preparednessrequirementinsectinc3.4fthetraccriteriaandchecklist, 7 thisdcumentservestw purpses.first,itprvidesanverviewftheplicies,prcedures,resurcesandcntractsthatenable HathiTrusttaddressthechallengesandthreatsendemictthefieldfdigitalpreservatin.Materialis therefreciteddirectlyfrmthehathitrustwebsite( versinfhathitrust sreviewfitscmpliancewiththeminimumrequiredelementsfthetrac CriteriaandChecklist, 8 andrelevantliteratureprvidedbykeyvendrsandserviceprviders. 9 Secnd, thisreprtexamineshathitrust scurrentlevelfdisasterpreparednessanddefinescurrentand frthcmingeffrtsinitsdevelpmentfadynamicandpractivedisasterrecveryprgram.perthe recmmendatinsfthetraccriteriaandchecklist,thisdcumentrecrdsthemeasuresand precautinsalreadyinplaceinregardst specifictypesfdisasters thatculdbefallhathitrust.these eventsincludehardwarefailure,datalss,netwrkcnfiguratinerrrs,externalattacks,creutility failure,frmatbslescence,sftwarefailure,physicalsecuritybreach,andmanmadeaswellasnatural disasters.whileafrmal,writtenplandetailingindividualrlesandrespnsibilitiesintherepsitry s respnseteachfthesescenarisisstillfrthcming,theevidencegatheredinthisreprtrevealsthat crucialelementsfadisasterrecveryplanarealreadyinplacewithinhathitrust. 10 EssentialHathiTrustBusinessFunctins AsthedevelpmentftheDisasterRecveryPlanprceeds,itisimprtanttbearinmindthat itsgalisntmerelytherestratinfhardwareanddatabutalstherecveryandcntinuityf essentialrepsitryfunctins.thefllwinglistrepresentscrefunctinsthatneedtbeaddressedby HathiTrust sdisasterrecveryplanandassuchshuldntbecnsideredacmprehensive representatinftherepsitry sfunctins.bydirectingplanningeffrtstwardspecificfunctins (ratherthantherganizatin sactivitiesasawhle),hathitrustmaypriritizeandfcusitsrecvery respnsesandresurcestensurethatthemstessentialfunctinsgbacknlinefirst.subsequent discussinfdisasterrecverystrategiesandriskmanagementslutinsinthisreprtarepresented undertheassumptinthatthecntinuityfthesefunctinsisaprimarybjective.thepriritizatinf thesefunctinsremainstbedeterminedbyanapprpriateauthrity Repsitryhassuitablewrittendisasterpreparednessandrecveryplan(s),includingatleastneff sitebackup fallpreservedinfrmatintgetherwithanff sitecpyftherecveryplan(s).therepsitrymusthavea writtenplanwithsmeapprvalprcessfrwhathappensinspecifictypesfdisaster(fire,fld,system cmprmise,etc.)andfrwhhasrespnsibilityfractins.thelevelfdetailinadisasterplanandthespecific risksaddressedneedtbeapprpriatettherepsitry slcatinandserviceexpectatins.fireisanalmst universalcncern,butearthquakesmayntrequirespecificplanningatalllcatins.thedisasterplanmust, hwever,dealwithunspecifiedsituatinsthatwuldhavespecificcnsequences,suchaslackfaccessta building. OCLCandCRL.TrustedRepsitriesAudit&Checklist:CriteriaandChecklist(2007)p HathitrustDigitalLibraryReviewfCmpliancewithTrustwrthyRepsitriesAudit&Certificatin:Criteriaand ChecklistMinimumRequiredElements,revisedMay20,2009.Availableat 9 CntactinfrmatinfrrelevantUniversityfMichigandepartmentsandserviceprvidersaswellasfrexternal vendrsmaybefundinappendixa. 10 AlistfresurcesrelatedtdisasterrecveryandtheplanningprcessmaybefundinAppendixD(Anntated ListfDisasterRecveryPlanningResurces). 11 ThislistfessentialHathiTrustbusinessfunctinswasdevelpedincnjunctinwithJeremyYrk

10 Ingest Ingestdigitalbjects(SIPs)viaGRIN thegglereturninterface(ra mdifiedingestprtalfrlcalcntent) ValidateingestedcntentwithGROOVE thegglereturnobject Oriented ValidatinEnvirnment(ramdifiedversinfrlcalizedingest) ArchivalStrage Preserveindefinitelydigitalbjectsandmetadata(AIPs)intheSharedDigital Repsitry(includesensuringtheintegrityandauthenticityfmaterials).This functinaddressestheneedsfpartnerlibrariesaswellasindividualusers. Recrdchangestandactinsnitemswhiletheyareintherepsitry Maintainapersistentbjectaddressfritemswithinrepsitry Disseminatin Prvideaccesstdigitalbjectsfrusers Allwfrthetextsearchesthrughavarietyffields Enablelargescalefull textsearches Permitthecreatinfpublicandprivatecntentcllectins Disseminatedigitalbjects(DIPs)tusers(viathepage turneraccesssystem anddataapi) DistributedatasetsandHathiTrustAPIstdevelpers ResearchanddevelpadditinalapplicatinsandresurcesfrHathiTrust Administratin Prvidetransparentandup t dateinfrmatintusersandthegeneralpublic viahttp:// Cmmunicateinfrmatinandcrdinateactivitiesamngstpartnerlibraries andhathitrustbardsandcmmittees. DataManagement UpdateandmanagetheRightsandGeIPdatabases BuildandmaintainCllectinBuilderandLargeScaleSearchSlrindexes Determineapprpriateuseraccessttextsviadatabasequeries SynccntentwiththeIndianaplissiteandbackupcntentttape

11 HathiTrust sdisasterrecverystrategies BasicRequirementsfrDisasterRecvery RyTennanthasidentifiedthreerequisitecmpnentsfadigitalDisasterRecveryPlan:(1) theusefaneffectivedataprtectinsystem(i.e.raid),(2)redundantpwerandenvirnmental systems,and(3)regularbackupfinfrmatinttapeand,ideally,taremtemirrredsite. 12 HathiTrusthasincrpratedalltheseelementsintitsdesignandperatin.ItsIsilnIQstragecluster prvidesahighdegreefdataredundancywithitsn+3parityprtectin;themichiganacademic CmputingCenterprvidesfullyredundantpwerandenvirnmentalsystemsfrHathiTrust infrastructure;andnightlytapebackupsandthereplicatinfdatatafullyperatinalmirrrsite lcatedatindianauniversityinindianapliswiththesamelevelsfpwerandenvirnmental cnditiningprvidemultiplecpiesaswellasgegraphicdistributinfcntent. HathiTrustisintendedtprvidepersistentandhighavailabilitystragefrdepsited files.inrdertfacilitatethis,theinitiative stechnlgycncentratesncreatinga minimumftwsynchrnizedversinsfhigh availabilityclusteredstragewithwide gegraphicseparatin(thefirsttwinstancesfstragewillbelcatedinannarbr, MIandIndianaplis,IN),aswellasanencryptedtapebackup(writtentandstredina separateannarbrfacility). Eachfthesestragertapeinstancesisphysicallysecure(e.g.,inalckedcageina machinerm)andnlyaccessibletspecifiedpersnnel.eachseparatestrage systemisalsequippedwithmechanismstprvidemirrredmanagementandaccess functinality,andemply100%dataredundancyinaneffrttpreventdatalss. 13 DetailsnparityprtectinandtheHathiTrustserverenvirnmentareavailablebelw(seeScenari1 andscenari5,respectively). DisasterRecveryStrategy#1:RedundancybetweentheAnnArbrandIndianaplisSites HathiTrust'sfirstlinefdefenseintheeventfadisasterisitshtmirrrsiteinIndianaplis. WhileingestfmaterialisrestrictedttheAnnArbrlcatin,bthsitespssesstwwebservers,a MYSQLdatabaseserver,andanIsilnIQstragecluster(currentlycmpsedf21 ndes, servers cmpsedfcentralprcessingunitsaswellasstrage).duringnrmalperatins,thisarrangement allwshathitrusttbalanceahighvlumefwebtrafficacrssbthsitessuchthatindividualuser requestsmaybehandledbyeithersiteinatransparentmanner.shuldthetlerancesfrfailurebe exceededatasite(asinadisastersituatin)thefailvercapabilitybuitintthehathitrustarchitecture enablestheremainingsitetprvideaccesstthedesignatedcmmunitywithutnticeableservice disruptins.asntedinthemay2009hathitrustupdate,withthefullperatinfbthlcatins, We arenwensuringthatusersdntfeeltheeffectsfsingle siteutages,suchasrutinemaintenance, 12 Tennant,Ry. DigitalLibraries:CpingwithDisasters. LibraryJurnal,15Nvember2009.Retrievedfrm 13 HathiTrust. Technlgy retrievedfrmhttp://

12 bytakingadvantagefsiteredundancy. 14 Hwever,becauseingesttakesplacenlyinAnnArbr,the lssfkeycmpnentstherewuldinhibittherepsitry sabilitytacquirenewcntent. HathiTrustutilizesIsilnSystem ssynciqapplicatinsftwaretsynchrnizedataatthe IndianaplissitewithnewlyingestedrupdatedmaterialfrmtheAnnArbrsite.Thesynct Indianaplisrunsn24separatesubsetsfthedataandeachnerunsevery2hurs,withthe exceptinfsundays.intherwrds,subset1runsatmidnightnmnday,subset2runsat2a.m.,and sn.themaximumtimefrdatatbereplicatedfrmannarbrtindianapliswuldtherefrebe threedaysplustheruntimefthesyncprcess(whichtendsttakelessthanthreehurs.) 15 SyncIQisanasynchrnusreplicatinapplicatinthatfullyleveragestheunique architecturefisilniqstragetefficientlycpydatafrmaprimaryclustertne lcatedatasecndarylcatin. 16 Allndes[ inbththesurceandtargetisilniqclusters]cncurrentlysendand receivedataduringreplicatinjbsinrealtime,withutimpactingusersreadingand writingtthesystem. 17 Arbustwizard drivenweb basedinterfaceisfullyintegratedint[ Isiln s prprietary]onefsmanagementtltcntrlallthefunctinality,including scheduling,plicysettings,mnitringandlggingfdatatransferredandbandwidth utilizatin. 18 Onlyfilesthathavechangedwillbereplicatedtthetargetclusters.Thiswillptimize transfertimesandminimizebandwidthused. 19 Intheeventthesecndarysystemisntavailableduetasystemrnetwrk interruptin,thereplicatinjbwillbeabletrllbackandrestartatthelastsuccessful cpyperatin. 20 Upnacriticalfailurerlssfnetwrkcnnectin,analertwillbesenttall recipientscnfiguredtreceivecriticalalerts. 21 DisasterRecveryStrategy#2:NightlyAutmatedTapeBackups HathiTrust sabilitytrecverfrmadisasterisalsensuredbythenightlyautmatedtape backupsperfrmedbythetivlistragemanager(tsm)clientapplicatininstalledntheingestservers cnnectedtthehathitruststrageclusterandmanagedbymichigan sitcstsmgrup.thetsm BackupServiceStandardServiceLevelAgreement 22 utlinesthebligatinsandrespnsibilitiesfbth theserviceprviderandhathitrust: 14 HathiTrust. UpdatenMay2009Activities (2009)retrievedfrm 15 Snavely,Cry(Head,UMLibraryITCreServices).Persnal n13July BackupandRecveryWithIsilnIQClusteredStrage, 2007p Ibid. 18 Ibid. 19 Ibid. 20 Ibid. 21 Ibid 22 PleaserefertAppendixF(TSMBackupServiceStandardServiceLevelAgreement)

13 TheprgressiveincrementalmethdlgyusedbyTivliStrageManagernlybacks upnewrchangedversinsffiles,therebygreatlyreducingdataredundancy,netwrk bandwidthandstrageplcnsumptinascmparedttraditinalmethdlgies basednperidicfullbackups. 23 ITCSisrespnsiblefrallfthecentralserverhardware,tapehardware,netwrking hardware,andrelatedcmpnents.itcsisalsrespnsiblefrhardwaremaintenance aswellassftwaremaintenance,administratin,andsecurityauditsnthecentral (nn client)tsmservers. (TSMBackupServiceSLA,sec.4.1) ITCSprvides7x24n callmnitringandsupprt,andstrivestkeeptheserversup inprductinatalltimes.thetargetup timeis99.9%fthetime.thetsmhardware designismdularandshuldallwusttakepiecesutfservicewithutaffecting custmers.wheneverpssible,systemmaintenancewillbeperfrmedduringstandard weekendmaintenancewindwsasdefinedbyitcs. (sec.4.2) then callstaff spagerinrealtime).(sec.4.6) ITCSisrespnsiblefrphysicalsecurity.Machineaccessaudits,OSsecurity,and netwrksecuritynthetsmserverendarealstherespnsibilityfitcs. (sec.4.9) Theservice[ ]includesdatacmpressin,dataencryptins,anddatareplicatin. (sec.1.0) ITCSwillmaintainatleasttwTSMsitesandwillmirrrdatabetweenthesitest prvideredundancyintheeventfadisaster.currentlythsesitesarethearbrlakes DataFacility(ALDF)at4251PlymuthRd.andtheMichiganAcademicCmputingCenter (MACC)lcatedat1000OakbrkDr. (sec.4.10) Bthfacilitiesaresecure,climatecntrlledsitesdesignedandbuiltfrhighavailable prductinservices. 24 Intheeventfacustmerdisasterwithlarge scale(afullserverrmre)datalss, ITCSwillwrkwiththecustmertptimizetherestretimetbestfurability.We willnlybeabletdevteresurcesttheextentthatthercustmersarent affected.restringlargefileservers(multipleterabytes)cantakeseveraldays.if custmerswanttminimizethisamuntftimetrestre,wecanpurchaseadditinal resurcesfrthispurpse.cntactusdirectly,andwe llwrkutascenariwith cstinginfrmatin.intheeventfamajorcampusutageaffectingalargenumberf custmers,itcsmanagementwillwrkwithcustmerstdeterminehwtpriritize custmerrestres. (sec.4.11) DisasterRecveryplanningistherespnsibilityfthecustmerunit. (sec.5.8) HavingestablishedthemainDisasterRecverystrategiesemplyedbyHathiTrust,wemaynwprceed tinvestigatethemeansbywhichitanticipatesandmitigatesthemstcmmnthreatsfacingdigital repsitries. 23 IBM. IBMTivliStrageManager:FeaturesandBenefits (2009)retrievedfrmhttp://www 01.ibm.cm/sftware/tivli/prducts/strage mgr/features.html?s_cmp=rnavn16june InfrmatinTechnlgyCentralServicesattheUniversityfMichigan. FrequentlyAskedQuestinsabutthe TSMBackupService (2009)retrievedfrmhttp://

14 Scenari1:HardwareFailurerObslescenceandDataLss Review:RisksInvlvingHardwareFailurerObslescenceandDataLss Thefllwingtablehighlightsthevariuseventswhichpsearisktthehardwareanddataf HathiTrust.Thesethreatsmaystemfrmflawsrmalfunctinsintheequipmentitselfrasaresultf externaleventsthatincludephysicalsecuritybreachesandnaturalrmanmadedisasters.the arrangementftheseptentialrisksreflectstherelativeseverityftheirrespectivecnsequences. Severity Highimpact MderateImpact LwImpact Event Lssatasinglepintffailure Anadditinalfailurepasttleranceswhennlynesiteisperatinal Serviceisunavailableandcanntberestreduntilcmpnentisrepaired/restred Failurefacmpnentpastredundancytlerance Systemnlngerhasredundancy:additinallssrfailurefcmpnentswill resultinlssfsystem.thisisaparticularprblemifnesiteisalreadydwn. Lssfdbserver(hmefRightsdb)rfbthWebserversatasitewillrender thatlcatininaccessible LssffurdrivesrndesineitherIsilnstrageclusterwillresultinthelssf thatinstance.theclusterwillbefflineandunablethandlereadrwrite requests;alltrafficwuldhavetbehandledbytheremainingsite. LssfUMArbrLakessitewuldpreventperfrmanceftapebackups. LssfUMMACCsitewulddepriveIUsitefdataredundancy Lssfingestserverswuldpreventnewcntentfrmenteringrepsitry Failurefredundantsystemcmpnents Includesredundantcmpnentswithineachsiteaswellasgeneralredundancy betweentheiuandumsites HTinfrastructurehasbeendesignedtavidsinglepintsffailureandt ensuredataandequipmentredundancy Servicecntinuesinanuninterruptedandtransparentmanner HathiTrust sslutinsfrhardwarefailureanddatalss ThethreatsfacedbyHathiTrust shardware(andassciatedapplicatinsaswellasthedata stredtherein)arecmprisedfthefailurefredundantfeatures,failurethatexceedscmpnents tlerancefrredundancy,andsinglepintsffailure.whilethefailurefredundantcmpnentsmay happenmrefrequently(i.e.,thelssfanindividualdrivewithintheisilniqcluster),suchlssesd nthavealargeimpactntherepsitry;eventswhichcmprmisesinglepintsffailurewillhave muchgreatercnsequencesfrthecntinuityfhathitrustperatins.atthesametime,whilea cmpnentmayhaveredundancynnelevel(frexample,therearefiveserversdedicatedtingest), thatcmpnentsimultaneuslymaybecnsideredatahigherleveltbeasinglepintffailure(i.e., becausetheingestserversarehusedinasinglechassis,theentireunitisvulnerabletaneventsuch asafire).thisdualityhighlightstheneedfrvigilanceandfresightinmanagingtherepsitry s infrastructure. BecauseHathiTrustreliesheavilyupnhardwaretfulfillitsmissinanddeliverservicestits designatedcmmunityfusers,theselectinfequipmentanddevelpmentfsystemarchitecture

15 hasaimedatminimizingthedangerspsedbysinglepintsffailurethrughtheintrductinf strategicredundancies.thebasicmeansfravidingthedisastruseffectsfhardwarefailurerdata lsshavebeentheestablishmentftheindianaplismirrrsiteandthenightlybackupfcntentt tape.(frmredetail,pleasereferttheprecedingsectin).whilethesestrategiesaccuntfr extrardinaryevents,hathitrust sserverreplacementscheduleallwstherepsitrytanticipatethe resultsfnrmalequipmentuseanddepreciatin.stepstsafeguardthelng termfunctinalityf HathiTrusthavetherefrebeencmplementedbyacnsideratinfbestpracticesfrdisaster preparedness. RedundantCmpnentsandSinglePintsfFailureintheHathiTrustInfrastructure ThefllwingsectinsprvideageneralutlinefHathiTrust sredundantcmpnentsand singlepintsffailure.giventhecmplexityftherepsitry sinfrastructure,unknwnr unanticipatedscenarismayexist;futuredisasterrecveryplanningwillthusinvlveaperidicreview fkeyfeaturesandvulnerabilities. SiteRedundancy:TheestablishmentfthemirrrsiteinIndianaprvidesHathiTrust withafullyredundantperatin.becausebthinstancesprvidefullaccesstcntent inadditinttherrepsitryfunctins,userswillntexperiencealssrdegradatin fserviceintheeventthatserviceislstfrmnesite.keyexceptinsthathitrust s siteredundancyarentedbelw. RedundantCmpnentsatEachSite:Thefllwingcmpnentsprvideeachsitewitha tleranceunderwhichlimitedfailureswillntdisruptmajrhathitrustfunctinsand userservices. Webservers:eachsitehastwserverssthatifnefails,thethermay cntinuethandletraffic.thesealshstthegeipdatabase. IsilnIQclusters:thecurrentcnfiguratinf21ndesfeaturesN+3parity prtectin;thisdataredundancypermitsthesimultaneusfailuref3drivesn separatendesrthelssfthreeentirendeswithutservicedegradatin. Ingestservers:theAnnArbrsitepssessesfiveserverssthatingestmay cntinue(albeitataslwerrate)intheeventfanyfailures. LargeScaleSearch(LSS)Slrindex:currentlyhusednthewebservers,butwill snbemaintainednfivenewserversinannarbr. SinglePintsfFailure: 25 Thesearecmpnentsfasystemwhich,iflst,willprevent theentiresystemfrmfunctining.eventhsecmpnentswithwhllyredundantpeer devices(suchasthewebringestservers)maybecnsideredsinglepintsffailureif theyhaveexceededtheircapacitytsustainlsses(i.e.,ifnewebserveratasitehas alreadybeenlst). SinglePintsfFailureattheCmpnentLevel:Becausenlynefthese cmpnentsexistsateachhathitrustsite,alsswillresultinsystemfailure. MYSQLdatabaseserver:husestherightsdatabase,ingesttracking database,andthecllectinbuilderslrindex Servernetwrkswitches Outbundnetwrkswitches SinglePintsfFailureattheSystemLevel:Whileanygivencmpnentmay havevariusdegreesfinternalredundancy(suchasmultiplepwersuppliesr 25 CntentinthissectiniscurtesyfCrySnavely(persnal frm13July2009)

16 multipledrives)itmightstillfailasawhleandthusresultinthelssfa particularinstancefhathitrust.thefllwingarecmpnentslcatedateach sitewhich,whilepssessedfinternalredundancies,arestillsubjectt cmpletelss(asintheeventfafire)andmaythusrenderasiteinperable. IsilnIQstragecluster:theentireclusterculdbelstinalarge scale event.additinally,thelssfafurthdriverndewillexceedthe cluster sfailuretleranceandresultinaservicedisruptin. Webservers:shuldnefail,theremainingserverwillbeasinglepint ffailure. Bladeserverchassis:sinceweb,ingest,anddatabaseserversarehused innechassis,theentireunitculdptentiallyfail. LSSindex:inthenearfuture,theserversinAnnArbrwillbethesle instancefthelargescalesearchindex. MirlyndatabaseandMirlyn2Slrindex 26 :thesearecurrentlykey cmpnentsftheumlibraryinfrastructure;shuldthesebe unavailable,accesstandusefhathitrustwillbecmprmised. KeyFeaturesfHathiTrust sisilniqclusteredstrage TheIsilnIQstrageclusterstresandprvidesdigitalbjectsfrHathiTrust spartnerlibraries andmembersfitsdesignatedcmmunity.theclusterprvidesahighdegreefinherentredundancy, whichgivesbthhathitrustsitesacnsiderabledegreeftleranceinregardstthefailurefvarius aspectsfthestrageunits.asneexample,isiln sprprietaryonefsperatingsystempermitsthe individualstragendes theindividualserversthatarethebuildingblcksfthecluster tfunctin as cherentpeers sthatanynende knws everythingcntainednthetherunitsinthecluster. Isiln'sOneFSperatingsystem[ ]intelligentlystripesdataacrssallndesina clustertcreateasingle,sharedplfstrage. 27 Becauseallfilesarestripedacrssmultiplendeswithinacluster,nsinglende stres100%fafile;ifandefails,alltherndesintheclustercandeliver100%fthe fileswithinthatcluster. 28 Adistributedclusteredarchitecturebydefinitinishighlyavailablesinceeachndeis acherentpeertthether.ifanyndercmpnentfails,thedataisstillaccessible thrughanythernde,andthereisnsinglepintffailureasthefilesystemstateis maintainedacrsstheentirecluster MirlynisthenameftheUniversityfMichigan scurrentonlinepublicaccesscatalg,whichissupprtedby thealephintegratedlibrarysystem.mirlyn2isabetaversinfum srecentlyimplementednextgeneratin catalg,basednthevufindplatfrm,whichwillbecmethemainlibrarycatalgnaugust3, IsilnSystems,Inc. IsilnIQOneFSOperatingSystem (2009)retrievedfrm 28 IsilnSystems. UncmprmisingReliabilitythrughClusteredStrage:DeliveringHighlyAvailableClustered StrageSystems (2008)p.7. Incmputerdatastrage,datastripingisthetechniquefsegmentinglgically sequentialdata,suchasasinglefile,sthatsegmentscanbeassignedtmultiplephysicaldevices.[ ]ifnedrive failsandthesystemcrashes,thedatacanberestredbyusingthetherdrivesinthearray. ( 29 IsilnSystems. BreakingtheBttleneck:SlvingtheStrageChallengesfNextGeneratinDataCenters (2008)p

17 HathiTrust sisilniqclustersensureahighdegreefdataredundancywiththeirn+3parityprtectin. N+3prvidestriplesimultaneusfailureprtectinsthatuptthreedrivesnseparateIsilnIQ ndes,rthreeentirendes,canfailatthesametimeandalldatawillstillbefullyavailable. TraditinalRAID 5parityprtectinresultsindatalssifmultiplecmpnentsfail prirtthecmpletinfarebuild.flexprtect,incntrast,autmaticallydistributesall dataanderrrcrrectininfrmatinacrsstheentireisilnclusterandwithitsrbust errrcrrectintechniquesefficientlyandreliablyensuresthatalldataremainsintact andfullyaccessibleevenintheunlikelyeventfsimultaneuscmpnentfailures. 30 Eachfileisstripedacrssmultiplendeswithinacluster,with[three]paritystripesfr eachdatablck. 31 ThefilesystemmayalsperfrmaDynamicSectrRepair(DSR)atthetimefanyfilewriting.Ifit encuntersabaddisksectr,thefilesystemwilluseparityinfrmatinelsewhereinthesystemt rebuildthenecessaryinfrmatinandrewriteanewblckelsewhereelsenthedrive.thebadsectr willberemappedbythedrivesthatitisneverusedagainandthewriteperatinwillbecmpleted. TheIsiln restriper isameta prcess/infrastructurethathasfurprimaryphasesthelp manageandprtectdataintheeventthatcmpnentsftheclustersustainapartialfailurer malfunctin.theprcessesrunasbackgrundperatinsanddntrequiresystemdwntime FlexPrtectrepairsdata(i.e.,intheeventfadrivelss)usingparity. IsilnOneFSwithFlexPrtectcanbasttheindustryleadingMeanTimet DataLss(MTTDL)frpetabyteclusters. 34 FlexPrtectintrducesstate f the artfunctinality,whichrebuildsfaileddisks inafractinfthetime,harnessesfreestragespaceacrsstheentirecluster tfurtherinsureagainstdatalss,andpractivelymnitrsandpreemptively migratesdatafffat riskcmpnents. 35 AutBalance rebalancesthedatainaclusteraccrdingtbusinessrules,inrealtime, nn disruptively. 36 Assnasthe[newrrepaired]ndeisturnednandnetwrkcablesare cnnected,autbalanceimmediatelybeginstmigratecntentfrmthe existingstragendestthenewlyaddedndeacrsstheclusterintercnnect back endswitch,re balancingallfthecntentacrssallndesinthecluster andmaximizingutilizatin IsilnSystems,Inc. IsilnIQOneFSOperatingSystem (2009)retrievedfrm 31 IsilnSystems. UncmprmisingReliabilitythrughClusteredStrage:DeliveringHighlyAvailableClustered StrageSystems (2008)p.7 32 IsilnX SeriesSpecificatins(prductbrchure) 33 InfrmatinntheIsilnrestripercmesfrmapersnal sentbyKipCranfrdfIsilnSystems,Inc.n1 June IsilnSystems. DataPrtectinfrIsilnScale OutNAS (2009)p.4 35 IsilnSystems,Inc. IsilnIQOneFSOperatingSystem (2009)retrievedfrm 36 McFarland,Anne. IsilnAcceleratesDeliveryfDigitalCntent TheClipperGrupNavigatr(2003). 37 IsilnSystems. TheClusteredStrageRevlutin (2008)p

18 Cllectcleansuprphanedndesanddatablckstpreventfragmentatinfdata. MediaScanverifiesdisksectrs. ThefunctinfMediaScanistscaneveryblckinthefilesystemlkingfr baddisksectrs.ifitencuntersabadsectr,itwillperfrmadynamicsectr Repair(DSR)anduseparityinfrmatinelsewhereinthesystemtrebuildthe necessaryinfrmatinandrewriteanewblcksmewhereelsenthedrive. MediaScanperidicallyreviewsdatablcksanddisksectrsthatmaynthave beenaccessed,frmafilelevel,inmnthsryearsandtherebyhelpstkeep thedrivesashealthyaspssible. AsftheOneFS5.0release,allfilesystemmetadatacanbecheckedbythe IntegrityScanrestriperphase.ThisprcesswillallwHathiTrusttcmpletelycheckfile dataandmetadataviaassciatedchecksums. Otherinstancesfinherentredundancyincludenn vlatileram,afullyjurnaledfilesystem,and sftwareapplicatinsthatmanageclientcnnectinsintheeventfande sfailure. OneFSisafully jurnaledfilesystemwithlargeamuntsfbattery backednnvlatilerandmaccessmemry(nvram)withineachnde,whichensurestheintegrity fthefilesystemintheeventfunexpectedfailuresduringanywriteperatin. 38 TheIsilnSmartCnnectmdule[ ensures]thatwhenandefailureccurs,allinflightreadsandwritesarehandedfftantherndeintheclustertfinishits peratinwithutanyuserrapplicatininterruptin.[ ]Ifandeisbrughtdwn franyreasn,includingafailure,thevirtualipaddressesntheclientswillseamlessly failveracrssalltherndesinthecluster.whenthefflinendeisbrughtback nline,smartcnnectautmaticallyfailsbackandrebalancesthenfsclientsacrssthe entireclustertensuremaximumstrageandperfrmanceutilizatin. 39 HardwareSupprtandService HathiTrustequipmentiscveredbysupprtandserviceagreementswithitsvariusvendrs (SunMicrsystems,Dell,CDW G,etc.).Agdexamplefnesuchagreementisfundinthe Platinum supprtprvidedbyisilnsystemsandwhichincludes: Extended24x7x365Telephne&OnlineHardwareandSftwareSupprt 24x7PractiveMnitring&Alerts Hme(frHardwareandSftware) ReturnPartstFactryfrRepairand4 hurreplacementpartsdelivery SupprtIQ(EnhancedServiceabilityDiagnstics)andSystemEventTracking On sitetrubleshting IsilnHardwareInstallatin SftwarePrductDcumentatin,ReleaseNtes,andaccesstPrductTechnicalNtes RemteDiagnsis(PrvidedUserGrantsAccess) Maintenance&PatchReleases 38 IsilnSystems. UncmprmisingReliabilitythrughClusteredStrage:DeliveringHighlyAvailableClustered StrageSystems (2008)p.9 39 IsilnSystems. DataPrtectinfrIsilnScale OutNAS (2009)p

19 MinrandMajrUpgradeReleases(IncludesPerfrmanceImprvements,New Features,ServiceabilityImprvements). 40 EquipmentTracking LITCreServices(CS)maintainsaninventryfserversnawikipageaccessibletitsstaff. Detailsincludeeachserver sname,lcatin,nlineandretiredates,upgrades,ntesnstrage,andits primaryservice.additinalinfrmatinisprvidedrelatedtspecificatins,supprtcntracts,andkey cntactinfrmatin.thecsserverinventryiscurrentlyutfdate. HardwareReplacementSchedule HathiTrustreplacesstrageregularly,apprximatelyevery3 4yearsrastheusable lifefstrageequipmentdictates (HTTRACC1.7) HathiTruststaffupgradehardwarenaregularbasis(i.e.,everythreerfuryears), andthelpdetectmrerapidgrwthindemands,thewebserverandstrage infrastructureshavetheirwnperfrmancemnitringthatindicateverlad cnditins. (HTTRACC1.10) TimelinefrEmergencyReplacementfHathiTrustInfrastructure Shuldaseriuseventrequirethereplacementfpart(rall)ftheHathiTrusttechnical infrastructure,thefllwingtimelineprvidesageneralestimatefthetimerequiredtrder,ship, andinstallnewequipment.acursryreviewfthetimenecessaryfrhathitrusttrecverfrma majrdisasteratthemainannarbrrindianaplisdatacentersuggeststhatalargeeventculdidle aninstanceftherepsitryfratleastamnthandahalf.inadditinttheserversandswitches mentinedabve,criticalcmpnentsincludefur30apwerdistributinunits(pdus)perrackand furracksperdatacenterasfthiswriting. SubmissinfPurchaseOrders: Frrdersunder$5,000,theM PathwaysapplicatinallwstheUniversity Library sbusinessmanagertsendpurchaserdersdirectlytvendrs. Frrdersver$5,000,PrcurementServicesnrmallytakesnettw businessdaystapprvethepurchase,buttheprcessmaytakeuptaweekif questinsariseradditinalpurchaseinfrmatinisneeded. DeliveryfEquipment: Prductsthevendrhasinstckandavailablefrimmediateshipmenttake1 3 daystbedelivered. Itemsthatneedtbecnfigured(suchasservers)usuallytake1 2weeks. Isilnstragewilltake3weekstbedeliveredinawrstcasescenari. Installatin: 3daysFTEfrIsilnIQclusterinadditintthetimerequiredfrtherservers, switches,pdusandrackunits. 40 IsilnSystems. SupprtAdvantageOfferings (2009)retrievedfrm

20 DataRestratin:abut.5TB/hur(15days,asfJune2009) 41 WhileHThasabut110TBfdatainitsstrage,thebackuptapesmaintained bythetsmgrupcntainrughly176tbfinfrmatinduetthedata encryptinusedtprtecttheintellectualrightsfthematerial(asf06/2009). Thelengthftimerequiredfra bare metalrestratin willbeinfluencedby tapemunts,netwrkspeed,restringtthenfsshares,decryptin,etcetera. Ifthelibrary/HTweretpurchaseanadditinaltapedrive(atrughly$20,000), theprcessculdbespedup,perhapstabut1tb/hur. Intheeventfalarge scaledisasterinwhichmultiplecampusunitsrequire extensivedatarestratin,thetsmbackupserviceslastatesthat ITCS managementwillwrkwithcustmerstdeterminehwtpriritizecustmer restres. (sec.4.11)thisdeterminatinwillreflecttheuniversityfmichigan s rganizatinalpririties 42 : Pririty1:Healthandsafetyffaculty,staff,students,hspitalpatients, cntractrs,renters,andanytherpeplenuniversitypremises. Pririty2:Deliveryfhealthcareandhspitalpatientservices Pririty3:Cntinuatinandmaintenancefresearchspecimens, animals,bimedicalspecimens,researcharchives. Pririty4:Deliveryfteaching/learningprcessesandservices Pririty5:SecurityandpreservatinfUniversityfacilities/equipment. Pririty6:Maintenancefcmmunity/Universitypartnerships. Fractinalrestreswuld,frthemstpart,runatcmparablespeedsunlesstherewas aneedtrestrealargenumberfrandmfiles,inwhichcasetherewuldbea decreaseinspeedduettapeseekandmunttimes. DelaysinrecveryculdbeincreaseddramaticallyiftheMACCdatacenterrits infrastructurehassustaineddamageandneedsrepair. HathiTrustandInsuranceCverageattheUniversityfMichigan TheOfficefFinancialOperatinsreviewsandaddsfinancialassetsgreaterthan$5,000tthe assetmanagementsystemftheuniversityfmichigan.theprpertycntrlofficeisthenrespnsible frtaggingfinancialassetswithuniqueuniversityfmichiganidentifiersandtrackingthem.risk ManagementServicesadministerstheUniversity sprpertyinsuranceandwillprvidethe reimbursementfreplacementcstsfritemsself insuredbymichigan.asfjuly2009,thenatureand extentftheuniversityfmichigan sinsurancecveragefrhathitrusthardwareremainedunder review.themaincntactwithriskmanagementservicesinthismatterhasbeencyndimesa,headf UMLibraryFinance. 41 Hanver,Camern(ITCSTSMGrupStrageEngineer).Persnal n23June UniversityfMichiganAdministrativeInfrmatinServices. EmergencyManagement,BusinessCntinuity,and DisasterRecveryPlanning (2007)retrievedfrmhttp:// n6july