Print-to-Mail Continuity Lessons Learned Jerry Montella, Vice President of Mail-Gard Client 1: Financial Institution $22 Billion Length of Contract: Data Hot Site Contract: Print/Mail Recovery Contract: Data Transmission Method: Contracted Recovery Requirements: Needed for Disaster: 2 yrs 8 months In Place thru Data Provider Direct Drive One Printer 48,000 images per day 3 Printers 6 Inserters 335,000 images per day Last Test Date: June 6, 2005 Test Scenario: Report Printing Only Data Processing handled by a 3 rd Party Vendor Declared 9/1 - Data Transmitted 9/13 - Recovered 10/5 Client 2: Public Sector Government Agency Length of Contract: Data Hot Site Contract: Print/Mail Recovery Contract: Data Transmission Method: Contracted Recovery Requirements: Needed for Disaster: Last Test Date: 1 Year In Place Thru Data Provider Channel Extension 1 Printer 1 Inserter 50,000 images per day 2 Printers 1 Inserters 265,000 images per day Never Declared 9/9 - Data Transmitted 9/13 - Recovered 1/18 1
Client 3: Financial Institution $6 Billion Length of Contract: Data Hot Site Contract: Print/Mail Contract: Data Transmission Method: Contracted Recovery Requirements: Needed for Disaster: Last Test Date: 1 Month In Place Direct with MG Direct Drive 1 Printer 1 Inserter 50,000 images per day 2 Printers 1 Inserters 180,000 images per day Had only recently signed a contract, no time to test Declared 9/1 - Data Transmitted 9/9 - Recovered tbd Client 4: Healthcare/Insurance $215 Million Length of Contract: Data Hot Site Contract: Print/Mail Contract: Data Transmission Method: Contracted Recovery Requirements: Needed for Disaster: 2 yrs In Place Direct with MG Direct Drive (declaration) 3 Printers 2 Inserters 150,000 images per day 1 Printer 1 Inserter 90,000 images per day Last Test Date: May 23, 2005 Test Scenario: All Applications Declared 10/26 - Data Transmitted 10/27 - Recovered 11/4 LESSONS LEARNED: Print-to-Mail Recovery Best Practices Really Do Make the Difference 2
Print-to-Mail Recovery Best Practices 1 Make sure all mission critical print-to-mail equipment and applications are included in your plan and covered in your contract 2 Test you plan ALL OF IT 3 Plan for the worst case scenario Best Practice # 1 Make sure all mission critical print-to-mail equipment and applications are included in your plan and covered in your contract Plan for everything you need or you re putting your company at risk! RISK OF NOT RECOVERING RISK OF NON-COMPLIANCE HOW LONG CAN YOUR COMPANY SURVIVE WITHOUT CASHFLOW? Best Practice # 1 Plan for and include all mission critical equipment and applications Client 1 Covered only a small portion of their print volume (reports) Contracted for only one printer Client 2 Though they had their volumes covered 3
Best Practice # 1 Plan for and include all mission critical equipment and applications Using 3 rd Party Vendors? (Data Processing, Data Recovery, Print/Mail) Know who is responsible for the DR Plan Know your role Compliance is your responsibility, not your vendors! Test, exercise, practice, trial, assessment Whatever you call it DO IT! The more you test, the more comprehensive your tests are the smoother your recovery will be Client 2 Had print/mail in the DR plan but - Never tested - On-Site Personnel did not know the plan details - Had no documentation Details Overlooked: - Start commands - Resources not identified - What job took what form - Page/form definitions - No print samples to verify positioning? 4
Combine your data recovery test and print/mail test Keep recovery providers informed of new and changing hardware and software requirements Test all aspects of your plan Client 3 Action: Reaction: Never tested print and mail with data recovery Data received 8 days after declaration Last Hot Site Test What are your recovery time objectives? What did you recovery? (systems, applications) Was your test successful? Did you meet time objectives? Client 1 Action: Reaction: Recent system change, hot site provider unaware work around solution was required Major delay in data transmission No Verification Processes: - what file was being transmitted? - how many pages to each file? - how many files transmitted? RISK?? Compliance requires the ability to validate!! 5
Lesson Learned Choose and efficient method of data transmission Host to printer (Direct Drive/Direct Connect) not the best option Can disrupt and lengthen recovery time Errors during print? - Recovery site must contact data site to correct the issue - Job must re-start from beginning - Time and paper is wasted Host-to-Host preferred Client 1 Client 2 Client 3 Action: Reaction: Material requirements & Postage were an afterthought Delay in recovery time - Cost Special forms: Envelopes: Postage: White paper used until stock arrived or job delayed Blank envelopes used or job delayed How to transfer funds to recovery center Meter? Indicia? Pre-sort requirements? Test Regularly Don t forget the details Test Completely and Document everything 6
Best Practice # 3 Plan for the Worst Case Scenario People Bench Strength Back-Up Staffing Plan Best Practice # 3 Plan for the Worst Case Scenario Client 1 Client 2 Client 3 People Few on-site personnel were operations employees Special circumstances meant distractions Most company personnel left, leaving operations to print/mail recovery staff Confidence is great - but not a best practice! Best Practice # 3 Plan for the Worst Case Scenario Bench Strength Provide Plan for Everyone Plan on a Worst Case Scenario 7
Summary Best Practices Make the Difference # 1 - Make sure all mission critical print-to-mail equipment and applications are included in your plan and covered in your contract If you don t include it all how can you fully recover? In the event of multiple disasters recovery vendor may not be able to support non-contracted equipment and applications Best Practices Make the Difference # 2 Client 1 Client 2 Client 3 Plan: Testing: Recovery: Plan: Testing: Recovery: Plan: Testing: Recovery: Incomplete Partial Possible due to excess capacity Longer than necessary Incomplete None Start up extremely delayed Longer than necessary Complete None Start up delayed 8
Best Practices Make the Difference # 3 Plan for the Worst Case Scenario Bench Strength/Back-Up Staffing Plan Communicate the Plan to Everyone Be prepared for work-around Rely on the experts to help make decisions Best Practices Make the Difference 3 Katrina recoveries would have been more efficient If: (easier, faster cheaper) Planned for and contracted for what they needed Process that were test and fully documented People who were trained/informed/had ownership of print/mail processes Best Practices Make the Difference Need More Proof? Client 4 DR Plan included all necessary equipment and critical applications Tested all applications Back up staff, back up documentation Planned for the Worst 9
Best Practices Make the Difference Client 4 More Proof? Declared 10/26 Data Transmitted 10/27 Recovery preparations at the recovery center began BEFORE the client arrived Staff arrive, print operations began 10/28 A flawless recovery execution! LESSONS LEARNED: Print-to-Mail Recovery Best Practices Really Do Make the Difference Questions? 10