Computer systems failure contingency planning

Transcription

1 Computer systems failure contingency planning Second Edition Cihan Cobanoglu, Ph.D., Stephen Blasik, and Mark G. Haley, CHTP A Technology Primer Developed by the American Hotel & Lodging Association s Technology and E-Business Committee Funded by the American Hotel & Lodging Educational Foundation NEI02ENGE PP-3465

2

3 This guide was written by Mark Haley, revised and updated by Cihan Cobanoglu, Ph.D., Stephen Blasik, and Mark Haley and developed by the Technology and E-Business Committee of the American Hotel & Lodging Association (AH&LA), with a grant from the American Hotel & Lodging Educational Foundation (AH&LEF). Technology and E-Business Committee Co-Chair Co-Chair Vice Chair AH&LA Officer Liaison AH&LA Officer Liaison AH&LA Staff Liaison Daniel Connolly, Ph.D., Associate Professor University of Denver Clay Dickinson, Executive Vice President Jones Lang LaSalle Hotels Cihan Cobanoglu, Ph.D. University of Delaware David Kong, President & CEO Best Western International Pam Inman, EVP & COO American Hotel & Lodging Association Laurence Barron, Vice President & CIO American Hotel & Lodging Association Members James O. Abrams ISHAE Representative President & CEO California Hotel & Lodging Association Laurence Barron Vice President & CIO American Hotel & Lodging Educational Institute Stephen Barth President & Founder HospitalityLawyer.com Kathleen Pearl Brewer, Ph.D. Professor/Director of Graduate Studies University of Nevada at Las Vegas Dennis Carpenter, MHS Director, Association Alliances Heartland Payment Systems Cihan Cobanoglu, Ph.D., CHTP Associate Professor University of Delaware HRIM Patton Conner Vice President, Information Resources Marriott International Daniel Connolly, Ph.D. Associate Professor University of Denver Clay B. Dickinson Executive Vice President, Strategic Advisory Jones Lang LaSalle Hotels Gregory A. Dugal ISHAE Representative Executive Director Maine Innkeepers Association Rich Ehlers Director ibahn Bernard Ellis Managing Director Americas IDeaS Revenue Optimization William Folkerts Owner Comfort Inn Mark Haley, CHTP Partner The Prism Partnership 2009 American Hotel & Lodging Association. All rights reserved. iii

4 Pam Inman, IOM, CAE Executive Vice President & Chief Operating Officer American Hotel & Lodging Association David Kong President & CEO Best Western International Monika Nerger Vice President of Technology, The Americas Mandarin Oriental Hotel Group Kirby D. Payne, CHA President HVS/American Hospitality Management Company Laura Powers, MHS Director of Business Development Teledex Corporation Tina Reese Regional Vice President First Data Commercial Services Revenue Sharing Alliances Jeffrey Senior Executive VP, Marketing & Sales Fairmont Raffles Hotels International Jules Sieburgh, CHTP O Neal Consultants Kevin H. Smith EVP & General Manager New Yorker Hotel Management Co., Inc. iv 2009 American Hotel & Lodging Association. All rights reserved.

5 Acknowledgements American Hotel & Lodging Educational Fondation s Research and Project Funding Committee Editorial Reviewers Pearl Brewer, Ph.D., UNLV Kutay Kalkan, University of Delaware Disclaimer This guide is intended only as a general guide concerning Hotel Computer Systems Contingency Planning and does not purport to be, nor should it be used as, a complete description of Hotel Computer Systems Infrastructure problems or solutions. Companies should not rely upon this guide for other than general information and should consult their employees and attorneys before implementing any suggestions or procedure or using ant forms contained in this guide. AH&LA does not warrant the accuracy of the guide, the accuracy of completeness of the procedure described in this guide, the effectiveness of such procedures or the effect of any forms contained herein American Hotel & Lodging Association. All rights reserved. v

6

7 Table Of Contents General Overview... 1 Purpose of This Document... 1 Why Computer Failure Contingency Planning is Important... 1 General Strategies in Computer Failure Contingency Planning... 3 Prevention & Minimization... 3 Physically Secure Data Centers... 3 Protected Electrical Supplies... 3 Preparation... 4 Contingency Plan & Testing... 4 Hardware and Software Inventory... 4 Alternative Locations for Contingent Operations... 4 React... 5 Reliable Current Data... 5 Increased Staffing Requirements... 7 Processes for Contingency Operations... 7 Recovery Process... 8 Applicable Case Studies... 9 Contingency Team Contacts Identification of Members Vendor Support Contacts Support Agreement Terms Escalation Path Program and Data Back-Up Locations of Back-Up Tapes Back-Up Software and Hardware Documentation Contingency Plan Documentation Essentials to Quality Documentation Locations of Vendor Documentation on System Operation PMS Interfaces Point-of-Sale System In-Room Entertainment Interface American Hotel & Lodging Association. All rights reserved. vii

8 Call Accounting System PBX/Voice Mail Interface Other Servers & Peripherals Guestroom Locking Systems Contingency Operations Major System Failures PMS Outage (Less Than Four Hours Estimated) Short-Term PMS Outage (More Than Four Hours, Less Than Two Night Audits) Long-Term PMS Outage (Two or More Night Audits PMS Recovery Processes and Testing Testing Point-of-Sale System CAS Call Accounting System Location and Contents of Contingency Supplies Crash Kits Importance of Testing Appendix Section A: Crash Kit Contents (Contingency Supplies) A.1 General Information A.2 Credit Card Supplies A.3 Forms A.4 Extra Office Supplies A.5 Miscellaneous Items Section B: Sample Control Sheet Formats B.1 Local Emergency Contacts B.2 Names, Titles, Roles, Contact Information B.3 Vendor Support Agreements B.4 Arrival/Departure Control Sheet B.5 Posting Control Sheet B.6 Room Revenue Control Sheet B.7 Checkout Control Sheet Front Office Cashier s Report B.8 F&B Cashier s Control Sheet B.9 Room Rack Sheet Works Cited... 31

9 GENERAL OVERVIEW Purpose of This Document The AH&LA Technology Committee commissioned this document, an informative guide identifying essential topic areas assisting hotels in preparing and planning for brief, short and long-term failures of their vital computer systems, especially associated with the Property Management System (PMS), Guestroom Locking System, Telephone System (also called Private Branch Exchange, PBX) and Point of Sale Systems (POS). Today, the majority of hotels are entirely dependent on these systems in order to deliver essential guest The survey addressed the level of corporate readiness for disaster recovery, concluding that only 12 percent of enterprises have an effective disaster recovery plan. Fully 83 percent of companies were ineffectively prepared for a technology disaster. Hempell & Wendland services, and relatively few of today s hotel employees have experience in working in a non-computerized hotel. Therefore, the Committee identified a need for a basic-level manual on how to prepare for and plan around a system failure. Computer Failure Contingency Planning is actually a sub-set of a broader discipline, Business Continuity Planning. Should the AH&LA membership find value in this document, a future publication may address the broader scope of Business Continuity Planning, which would encompass maintaining and restoring operations in the event of natural and manmade disasters (such as a hurricane or terrorist attack). Why Computer Failure Contingency Planning Is Important The scope of this manual mainly focuses on computer system failure in relation to the lodging industry, but limited to providing basic operating functions. The Contingency Plan includes multiple strategies, actions and procedures to resume business operations and functions associated with computer failure. Having a well-developed plan in place is essential for brief, short and long-term system failures. Similar to other important decisions in the hotel industry, priorities are focused on guest services, revenue and costs. If the hotel does not know which rooms are vacant and clean during a short-term failure of the PMS, then they will not be able to check guests in or service walk-ins. The lack of a plan thus decreases guest service levels and reduces revenue. Without a plan in place, if there is a long-term failure 2009 American Hotel & Lodging Association. All rights reserved. 1

10 of the POS, then one might have to close the restaurant. The lack of a plan decreases revenues and guest service, not to mention putting the F&B staff out of work, directly impacting employee satisfaction. A long-term failure of the PMS, such as a complete disk drive failure without adequate data backups, can easily put a hotel out of business by losing all of the in-house guest receivables, all of the accounts receivable, future reservations and future availability. With a tested plan in place, the hotel should be able to survive these risks sufficiently if not gracefully. Without a plan, the hotel simply wouldn t know who owed how much, who had a reservation at what rate and whether or not they could take another reservation. This document discusses various elements and processes that provide guidance for IT systems restoration during a system disruptions or disasters, enabling the hotel to establish a coherent and effective contingency plan. These plans will serve the hotels as Users Manuals that effectively facilitate the recovery of IT systems, operations and data after a disruption, major or otherwise American Hotel & Lodging Association. All rights reserved.

11 General Strategies in Computer Failure Contingency Planning Every hotel and hotel company have unique circumstances according to their location, flag if any, range of services offered, systems installed and so on. Therefore, Contingency Planning must be done at the property level, specific to the needs of the property. While recognizing this fact, there are key strategies common to almost any Contingency Planning effort. This section reviews these general strategies, separated into the following key phases: Prevention and Minimization of failures Preparation for possible failures Responding to actual failures Recovery from failures Prevention & Minimization Time and effort spent on Prevention, Minimization and Preparation greatly mitigates the impact of a computer failure on guest services, hotel operations and hotel profitability, and accelerates the recovery process significantly. These conditions are true for almost any variety or duration of computer system failure. Physically Secure Data Centers Although in many hotels the Data Center is a closet behind the Front Desk, the prudent hotelier will recognize that it is still a Data Center and needs to be treated as such. Physically securing the room(s) containing critical systems can prevent and minimize a wide range of potential causes of failure. Data centers should be locked, accessible only from the back of the house and not have signage indicating the function of the room. Data centers should be windowless and not on exterior walls. Data Center should be located above grade and not have major plumbing directly above them to minimize water intrusion in case of a flood or plumbing failure. Data Centers should have independently controlled HVAC service. Larger Data Centers should have two sources of cooling, not just house HVAC. As discussed below, Data Centers and the equipment in them should have protected electrical supplies. Protected Electrical Supplies Prevention and Minimization Preparation Responding Recovery Many system outages revolve around transient or longer-term failures of clean, reliable electricity. Protecting computer systems with Uninterruptible Power Supplies (UPS), diesel generators and surge protectors will often go a long way to avoiding system failures altogether or make responding to them less problematic American Hotel & Lodging Association. All rights reserved. 3

12 Management must verify that all key systems are on UPS-protected power, including at least one terminal and printer on the PMS. If the electricity fails, the purpose of the UPS is to prepare for an orderly shut down and then to shut the systems down. Preparing for an orderly shut down includes running your Contingency Reports and notifying affected departments. Computer equipment not on a UPS should be turned off before the power comes back up. If the hotel has a diesel generator in addition to the UPS, it probably is not engineered to support the entire building s load, so a shutdown is in order even then. Preparation Contingency Plan & Testing Clearly, creating a Contingency Plan as described in this document is an essential part of preparing for a possible system failure of any severity. Likewise, testing the plan so that staff and management know how to implement it becomes vital to ensuring that the team knows how to implement it and recover from the failure. Hardware and Software Inventory The Plan should have an inventory of all of key hardware and software used in the hotel. While serial numbers of desktop PCs are not important to a restore effort, detailed information about all servers can be essential to have at hand. A key piece of inventory information often overlooked, record the make and model of all back-up devices and software. You will need this information if you need to send your back-up media to a remote site for recovery. Alternative Locations for Contingent Operations When creating your backup plan, take into account the workflow of the organization, department, or group that you re protecting. Ask yourself: am I protecting the right information at the right time? If at any point of failure can cause a workflow bottleneck, make sure that failure can be overcome quickly. If not, the costs of downtime will be multiplied by the number of people sitting around twiddling their thumbs. Cougias Many Contingency Plans call for moving operations to an alternate location if the use of the hotel or its Data Center is compromised due to an emergency condition such as fire, flood or terrorist attack. For many hotels, the alternate location will simply be a corporate office or sister property outside the affected area where a server can be installed and data tapes restored to it. A more elaborate approach, outside the scope of this document, is to contract with a service provider for hot site services, where they promise to make American Hotel & Lodging Association. All rights reserved.

13 available server capacity and workspace for you to move your operations to. Few hotels choose this costly type of alternate location solution. React Reliable Current Data Although we cannot overemphasize the value of reliable back-ups, in all but the most-sophisticated operations the back-up becomes out-dated in hours or minutes after it is taken and is not immediately available. This time lag establishes the need for reliable and more recent current data to know the current state of the house in time to deliver effective guest service. To use the PMS as an example, most PMSs have a group of reports that together can maintain operations. Some PMSs have a dedicated Contingency Reports menu; others allow the hotel to configure a job stream of selected reports that can be run as a batch. (Figure 1, screenshot of Reports Menu with Down-time Reports selected from OnQ. Image courtesy of Hilton Hotels Corporation) In reference to Figure 1, some PMSs generate Contingency Reports automatically according to a schedule defined by the hotel. These reports can be sent to print or saved as files on the PMS or another server. If the hotel is saving the files rather than printing them, then utilize another server to store them. If the reason you need the Contingency Reports is because the PMS server s disk drive failed, then you can t access them unless they are on another device. Of course, ensure that you have protected power to retrieve and then print the files in case the root problem is an electrical failure American Hotel & Lodging Association. All rights reserved. 5

14 (Figure 2, screenshot of configurable schedule for Contingency Reports. Image courtesy of Northwind.) These reports will contain, at a minimum: A list of all guests in-house, checked-out or expected today with balance, rate, group affiliation, special services and billing instructions, sorted in room number order. This report is used to maintain balances as charges are incurred during the outage. A list of all guests in-house, checked-out or expected today with group affiliation, comments and typically part of the address field, sorted in alphabetical order. This report is used to find out what room a given guest is in. A list of every room in the hotel with the current occupancy status: vacant and clean, vacant and dirty, occupied, out of order, #days available. This report is used to assign rooms to new arrivals and create housekeeping assignments In reference to Figures 1 and 2, the hotel may refer to these reports as Contingency Reports, Back-Up Lists or Downtime Reports. Most PMS vendors advise hotels to run current contingency reports hourly or as business activity dictates. This task is generally assigned to whoever is responsible for the Front Desk during each shift American Hotel & Lodging Association. All rights reserved.

15 We also recommend that the hotel run on an at least daily basis the following reports: All expected arrivals for next five days A room availability report for the next 90 days or longer, depending on the property s seasonality A blocked room report for the same period as above This task is generally assigned to the Reservations staff. Increased Staffing Requirements Manual operations always take longer and require more personnel to deliver acceptable service levels than when we have access to the automated tools we use every day to run hotels. Hotel managers should expect to call in additional staff during any extended period of downtime. Night Audit, Housekeeping room assignment, check-in, check-out and controlling room inventory all take longer and will require additional personnel during the system outage. Many of these human resources can be drawn from other departments during the emergency, especially Accounting. Processes for Contingency Operations Each hotel must have well-considered processes in place for dealing with the failure of key systems. These processes will vary according to the nature and expected duration of the failure. Brief, short and long term failures call for somewhat different approaches. For these purposes, define a brief outage as four hours or less, short-term as less than two night audit cycles and long-term as lasting two night audit cycles or more. A computer system failure affecting only a single system (a disk drive failure, for example) requires a different response than one affecting the entire hotel (a fire, extended electricity outage or other disaster, for example). Most system failures are fortunately of shorter duration and affect a single system only. Many hotels will determine that some transactions are not worth capturing manually during a system failure. Telephone calls represent a class of transactions that a hotel will sometimes choose to give away during a system failure. Note that for transactions involving revenue-share partners (such as In-Room Entertainment, typically), the partner will still expect to be paid even if your PMS was down, so the cost implications of ignoring those transactions is much greater than merely foregoing the hotel s commission. Another means of simplifying transactions and transaction accounting during a system failure is to simply alter the services offered. If the restaurant POS fails completely and will be down for several days, then a logical management decision would be to switch to buffet-only service with evenamount pricing, tax and tip included American Hotel & Lodging Association. All rights reserved. 7

16 Interface failures represent another class of situation to consider. Both the PMS and PBX may be functioning normally, but if the interface between the two is down, then contingency procedures must be in place to enable outbound telephone calls from newly-checked in rooms. If the interface between tpms and In-Room Entertainment system fails, then contingency procedures must call for enabling Pay-Per-View movie purchases in checked-in rooms and also for posting of movie purchases to guest folios. Guest communications represent another set of processes. For a brief PMS outage, periodically send a hotel manager into the lobby to explain to guests that the Front Desk system is temporarily down, we are operating normally but in a manual mode and although check-ins may take a little longer, we will get everyone checked in. Longer outages will call for a letter from the General Manager delivered to each occupied guest room and posted at the Front Desk. The specifics of these processes will be unique to each hotel s business circumstances and systems installed, but must be documented and staff trained on in order to ensure that guest services are of the desired level. The contingency operations processes suggested below in this document are based on the concept of using the Contingency Reports from the PMS before the system failed and manually maintained control sheets for various functions to maintain services during the system failure. The control sheets are essential to recovering to normal operation after the system is restored. Recovery Processes Running the hotel and delivering essential guest services is not sufficient. Contingency Planning must also include how to use the Contingency Operations processes to bring the system up-to-date with activities that occurred while the affected system(s) were down. Typically, when system services are restored, the data will be current as of the moment when the system went down or as of the most recent data backup. In either event, the Contingency Reports from before the system failed and the various control logs created during the system failure must be entered into the system in order to return to normal operations. System Backed Up to Tape System Fails Contingency Operations Using Contingency Reports and Logs System Up; Back Up Tapes Restored Data from Contingency Reports & Control Logs Input American Hotel & Lodging Association. All rights reserved.

17 Applicable Case Studies Hemphill & Wendland 1 present two examples highlighting the importance of contingency planning. One example, the World Trade Center bombing of 1993 was a terrorist attack. The other, Hurricane Iniki in 1992, illustrates the need for contingency operations due to natural disaster. They also summarize a study of 320 computer failures conducted by Comdisco, categorizing the cause of each: Hardware Problems 24% Power Outage 16% Hurricane 16% Flood 15% Miscellaneous 12% Fire/Explosion 7% Bomb 5% Earthquake 5% World Trade Center Bombing 1993 A case in point involved the bombing of New York s World Trade Center in The terrorist act inflicted physical damage to a large area of downtown Manhattan, including an adjacent hotel. Although the hotel was evacuated in less than fifteen minutes with no fatalities, the telephone and computer system was damaged. The hotel could not account for guests and employees. Communications failures prevented management from communicating with emergency services. The loss of heat could have caused further damage to existing systems from the bursting of frozen water pipes. The hotel s experience points to one of the key elements in a technology disaster recovery plan maintaining critical operational and financial data at an off-site storage facility. Following a disaster, off-site facilities that maintain reservation and accounting information would be available immediately for retrieval of in-house guest records, financial documents, guest history and sales information, minimizing a disaster s effect on a hotel s data and information. 1 Hempell, Christian E. and Wendland, Nicole R. (1999). Disaster Recovery in Hospitality The Risks in Computerization and Information Management. Retrieved February 3, 2009, from Hotel Online Web site: Recovery.html 2009 American Hotel & Lodging Association. All rights reserved.

18 Hurricane Iniki, 1992 One of the most destructive events for hospitality companies occurred in September 1992 as Hurricane Iniki unleashed winds up to 200 miles per hour on the Island of Kauai, Hawaii. The damage amounted to $1.6 billion and resulted in 85 percent of the island s hotel rooms shutting down for at least two months. With 30-foot waves and complete power outages, all hotel technology systems on the island were lost. Particularly in areas with potential for destructive weather, the need for disaster recovery and off-site back-up systems cannot be overstated. Hotel properties without technology disaster recovery plans may be closed for much longer periods of time than other companies American Hotel & Lodging Association. All rights reserved.

19 CONTINGENCY TEAM CONTACTS Identification of Members The Contingency Plan needs to identify the members of the Contingency Team. The Plan document should list them all by name, position and role in the response effort. Key roles include: Team Leader Responsible for implementing the overall plan and recovery process. Typically the highest-ranking hotel executive available, such as the General Manager or Manager on Duty. IT Leader Responsible for restoring service from the affected system(s). Typically an IT manager in a larger hotel; likely an outsourced IT resource or key operator in a smaller property. Department Heads Responsible for maintaining service levels in affected operating departments during the outage and restoring system data afterwards. Typically Front Office management in a PMS, PBX or locking system failure and Food & Beverage management in a POS failure. Accounting personnel have key functions in restoring the data maintained manually during the outage to their proper systems for revenue recognition. The Contingency Team will be a lengthy list of managers in a large or complex hotel or resort. The list must be maintained as people come and go, and must include multiple forms of contact information, including home and cellular telephone numbers, pagers, etc. All members of the team must be aware of their roles in responding to a failure. Hotels operated by a management company or a brand will need to include corporate contacts on their Contingency Team list, especially the Central Reservations Office. Please see table 1 and table 2 in Appendix Vendor Support Contacts The Contingency Plan should contain a list of key vendors and suppliers of technology products that will be of value in a failure. This list should include contact information for telephone support for all major vendors of critical systems, especially PMS, POS, PBX, etc. See sample table 3 in Appendix. Support Agreement Terms Most hotels have support agreements with the vendors of their missioncritical systems such as PMS, POS, Sales & Catering, etc. The standard agreements initially proposed by vendors typically impose little obligation on the vendors for response to emergency situations, such as a system outage. The standard support agreement will invariably exclude any vendor 2009 American Hotel & Lodging Association. All rights reserved. 11

20 responsibility for contingent liabilities, protecting the vendor from paying for losses consequential to the system failure. These losses might include lost room revenue due to inability to provide guest services, lost revenue from a failed posting interface, loss of goodwill and so on. Hotel companies should negotiate terms in their support agreement that require specific obligations to support the hotel in a system failure, for whatever reason. Typical terms will include: Definition of varying severity levels of system failures Maximum response times for telephone support for each severity level Defined Escalation Path for elevating the ownership of problem resolution within the vendor organization, going higher up as the problem severity or duration increases Minimum requirements for back-up devices and practices and perhaps power protection Vendors will reasonably negotiate to a responsibility to help you with the repair of the system failure. Many support services, especially those required due to a root cause not related to the vendor s software (flood or power problem, for example) may be billable services. They will not normally assume any responsibility for the recovery process, such as inputting data. Escalation Path A Contingency Plan must have a clearly defined Escalation Path to resolve an issue. The Escalation Path should provide a method to promptly resolve a difficulty. The Team Leader role, responsible for overall resolution and service delivery during the outage is a pivotal figure in escalating decisionmaking. The figure 2 on the following page is a structured model outlining the responsibilities and tasks that are needed to constitute a final decision made by an individual. 2 Collins & Cobanoglu American Hotel & Lodging Association. All rights reserved.

21 Identify Difficulty Consult among Contingency Team Agreement? Executive Decision Agreement? Manager Overview Resolution Attained 2009 American Hotel & Lodging Association. All rights reserved. 13

22 PROGRAM AND DATA BACK-UP Explaining the importance and need for reliable and tested back-ups of all programs and data are beyond the scope of this document. Suffice to say here that many system failures that call for the implementation of the Contingency Plan may require restoring data from back-up media. The General Manager of the hotel needs to assure herself that: Programs and data are backed up reliably according to a documented practice accepted by the involved vendors, brands and management companies At least two recent generations of back-up (i.e., last night and last week) are stored in a fireproof cabinet or safe outside of the hotel Rotate only the oldest tapes back into the hotel, so that there is always one copy off-site Back-ups are periodically verified Someone in the hotel knows how to restore data from back-up media reliably, at least with vendor assistance Restores are tested periodically to ensure that everything works and staff actually knows how to conduct a restore Locations of Back-Up Tapes Management has a primary responsibility to ensure that all data and programs necessary to restore operations after a failure are properly backed-up. Once the data and programs are backed up, at least two generations of the back-up media should be stored outside the hotel, off-site. Considerations in the selection of the off-site storage location include: Security Is your media secured from theft or loss in the off-site location? Cost How much will it cost to store and access your back-up media? Location Is the off-site storage location close enough that you can readily access the data when required? Environment Is the storage location temperature and humidity controlled One tactic some hotels have used is a reciprocal off-site storage agreement with another hotel in the area, where each stores the other hotel s media. Others store the tapes in a secure fire-proof located in the hotel, but in a separate wing or other location relatively remote from the Data Center. The recommended solution is to use an off-site records management service to store and transport tapes American Hotel & Lodging Association. All rights reserved.

23 Another approach is to back up across the Internet to a hosted back-up service, making the location of the back-up media a non-issue. Most experts would recommend using this approach only for incremental data backups, with multiple generations of program back-ups available on physically accessible media. There are a number of such service providers, with specialized software to facilitate the back-up and restore processes. These hosted service providers will typically utilize a Tier 1 co-location facility in a city remote from the hotel, which offers a major benefit to the hotel: If the cause of the outage is a major regional natural disaster (earthquake, for example), then having the data in another city is valuable. Back-Up Software and Hardware Tape drives remain the most common and effective approach to program and data back-up. Other options include optical media (CD/DVD) and Internet back-up services as described above. Regarding Internet back-up services, recognize that the speed of the download capability will govern the duration of a system restore: it could take several hours to restore the required back-up files. Also note the use of RAID ( Redundant Array of Inexpensive Disks ) arrays as an intermediate approach to data protection. Applicable in various levels, RAID arrays distribute the data and programs across multiple disks such that if one fails, the server can simply keep going without loss of service. Some RAID implementations can also offer performance benefits. Although RAID arrays can provide a significant layer of fault-tolerance, the only rational recommendation is add them to a daily back-up strategy, not in place of. Many software providers will specify the use of particular back-up software, especially PMS vendors. This practice enables the software vendor to assist in the restore process if needed. If you are using a back-up program that the vendor does not know, then you are on your own for a restore. Support agreements often specify the use of a specific back-up application American Hotel & Lodging Association. All rights reserved. 15

24 DOCUMENTATION Contingency Plan Documentation The Plan should be recorded in written form, with hard-copy printed versions readily available to hotel management. Soft-copy versions should also be available stored on a computer. A sample outline for a Contingency Plan document is in the Appendix. The Plan must contain enough information in enough detail so that current hotel management can maintain services as needed and that third parties can assist in the replacement of failed equipment. Ownership and maintenance of the Contingency Plan rests with the General Manager of the property. The Plan should be reviewed at least annually and updated with personnel and system changes. Essentials to Quality Documentation Good documentation is essential to an effective Contingency Plan. This means access to vendor-supplied documentation on utilizing the various programs and devices (such as application programs like the PMS and utility programs like your back-up software, hardware and media) but also data unique to your hotel. Include hard and soft copies of key parameter tables from major systems in the Plan. This would include the room number table, room type table and revenue account list from the PMS and menu table from the POS. In a worst-case scenario where no data backup is available at all, these records could at least provide a starting point for recovery. Like the Plan, this documentation should be stored in hard copy as well as soft copy files. If there is no electricity, the soft copy files stored on a computer will not be very useful. Locations of Vendor Documentation on System Operation Management should designate a particular location inside the hotel, to store reference copies of any necessary vendor documentation, particularly the back-up program and hardware, as well as any application software. PMS Interfaces A hotel may need to implement contingency operations because an interface between the PMS and another system is down. Both systems may be operating normally, but without the interface, the operation needs to adapt. Also, in the event of a PMS failure, the hotel will need to adopt contingency operations for the systems dependent on interfaces to the PMS American Hotel & Lodging Association. All rights reserved.

25 Point-of-Sale System Most POS systems support a number of settlement types (cash, Visa/ Mastercard, room charge, etc). Some or all of these settlement types will be interfaced to the PMS. Hotel POS screens should be configured with a settlement method that allows checks to be closed without going through the interface. The specifics will vary with the POS and its configuration. F&B personnel should use this settlement method when the interface or the PMS itself is down. F&B personnel must physically carry room charges or phone them in to the Front Desk for manual posting during contingency operations. Doing this promptly is essential during breakfasts, when many guests rush to check out immediately after breakfast. Many POS systems have a non-interfaced settlement method to close checks out to when the PMS is down. In-Room Entertainment Interface If the In-Room Entertainment (IRE) system interface is down, you will need to enable movie viewing for each room, much like the PBX status. Likewise, movie postings must be done manually, typically during night audit from a report generated by the IRE system. Some IRE systems will buffer charges until they sense the PMS interface is up. Then they will attempt to post movie charges to the PMS. This will often create discrepancies if the guest that watched the movie is not the one that was checked into the room in the system as of the time the PMS came back up. Call Accounting System If the Call Accounting System (CAS) interface fails, the hotel may choose to manually post all telephone charges, only high-value calls or to ignore them. Most CASs offer an option to print all calls as they are made or all calls since a given time. Using either method, designate someone to post the calls. Some PMS interfaces add taxes and surcharges to the amount reported by the CAS. If your systems are configured in that manner, then advise the posting clerk what markup to increase the amount posted by. As with the IRE systems, many CASs will buffer call charges until it senses the PMS is back up. PBX/Voice Mail Interface If the interface(s) between the PBX and/or Voice Mail system and the PMS fail, then as guests check-in to the hotel, the PBX operators must be advised to enable outbound calling for all guests that have established credit with 2009 American Hotel & Lodging Association. All rights reserved. 17

26 the hotel. This process may need to be repeated for some Voice Mail systems depending on the specific configuration. Most hotels will not bother to turn outbound calling off after a check out, but if you checked guests into the Voice Mail system, you must check them out in order to cancel their mailbox before the next guest checks in to that room. When the PBX and PMS detect that the interface is back up, they will initiate a handshake to synchronize the current room status correctly. Most interface configurations assume that the PMS is the authoritative source of data and PMS status will overwrite PBX status. Today s PBX systems do not fail very often. They are very reliable and usually feature built-in redundancy. When they do fail, it is likely to be a partial failure that is readily resolved by replacing a board. If a PBX completely fails or there is a fire in the telephone room, the only real recourse is to work with your vendor to get a new system installed as quickly as possible. A hotel that needs a new PBX on an emergency basis will likely be forced to close until telephone service is restored due to life safety and liability concerns, a good argument for maintaining PBXs under maintenance agreements. Other Servers & Peripherals Guestroom Locking Systems The basic approach to contingency operations for most guestroom locking systems is to maintain an inventory of contingency key cards, often called fail-safe keys by some lock system vendors. Make these keys in advance, label them clearly and store in a very secure place accessible only to the Manager on Duty, typically a safe deposit box at the Front Desk with the key on the MOD ring. Most locking systems support multiple levels of keys at any one time. The fail-safe level keys should work without re-phasing the lock if used in the proper order. When normal locking system operations are restored, the normal guest level keys should also function without re-phasing required. Making contingency keys is tedious and time-consuming. Most manufacturers recommend maintaining three complete sets of contingency keys, each to be used in sequence. Therefore, contingency keys must be labeled with the room number and sequence. Color-coded stickers may be of value to establish the sequence American Hotel & Lodging Association. All rights reserved.

27 CONTINGENCY OPERATIONS Major System Failures The discussions below begin with managing through a PMS outages of varying durations, the goes on to discuss failures of several other critical systems. The following section describes suggested contingency operations for brief, short-term and long-term system failures. Hoteliers often cannot identify that a given situation will turn out to be a brief problem (< four hours), a short-term failure or a long-term failure (two or more night audit cycles). What appears a minor problem creating a brief outage can often evolve into a longer-term issue. Other problems are obviously longer-term, such as a fire in the computer room. Management s first task is to assess the severity of the situation and estimate the duration of the problem. For an outage caused by some external factor (power failure, natural disaster, terrorist attack, for example), the obvious priorities are safety of guests and staff first, security of data back-ups second and salvaging hardware and physical assets third. Note that for many hotels during Hurricane Katrina the contingency operations consisted largely of clearing guests and staff from the hotel, securing the building and evacuating with tapes and servers to a safe location. The following sections discuss basic operational practices for managing through failures of interfaces and several major operational systems such as the PMS or POS. Hotels may find these practices useful to incorporate into their Contingency Plan documents and Plan tests. PMS Brief Outage (Less Than Four Hours Estimated) If the duration of the failure is estimated to be very brief, less than four hours: Alert the management team and affected departments Implement contingency operations for interface failures as discussed above, such as checking guests into PBX, settling POS charges to non-interfaced methods, etc. Secure all older copies of the contingency reports so they cannot be confused with the most recent reports Make photocopies of the most recent contingency reports if that is possible and distribute, especially to telephone operators and housekeeping Give a copy(ies) of the guest list by name to the telephone operators 2009 American Hotel & Lodging Association. All rights reserved. 19

28 Distribute radios to key personnel, especially if telephone service is affected. If not enough radios are available, designate individuals to hand-carry communications between the Rack Clerk, Housekeeping and PBX Get the crash kit(s) in case the contents are needed. Centralize the room inventory function with a single individual located at or near the Front Desk, essentially a rack clerk from when we used wire-frame room racks to control room inventory. This person will be responsible for assigning rooms for arrivals and coordinating room availability with Housekeeping. Update the Contingency Reports with all arrivals, departures, etc that have occurred since they were last run Registration cards and cashier copies of folios will be the best sources of this data Coordinate with Housekeeping to maintain the current status of room availability as rooms are cleaned, checked-out or checked in If availability for the day is tight, immediately get an accurate House Count and maintain it during the course of the outage As arrivals and checkouts are processed manually, cross them off the Contingency Reports so that the Contingency Reports reflect the current status of the house Record all transactions related to arrivals and departures chronologically on a separate control sheet (see Appendix for samples) This document, and similar ones, will greatly facilitate the recovery process As guests check-in, advise PBX to enable outbound calls Centralize the revenue posting function with a single individual located at or near the Front Desk, essentially a posting clerk. Utilize posting slips to record the charges and file them in folio buckets Separately record all room charges on a posting control sheet chronologically to facilitate recovery Separately record all checkout transactions onto a control sheet to facilitate recovery If there is a Central Reservations Office, refer reservations inquiries to them American Hotel & Lodging Association. All rights reserved.

29 If the hotel s PMS is tightly integrated to the Central Reservations System, the CRS may be able to provide availability and expected arrivals reports to the hotel during the system failure This capability becomes very valuable in extended system outages Short-Term PMS Outage (More Than Four Hours, Less Than Two Night Audits) If it appears that the failure may extend beyond four hours but, not more than one night audit, then implement the above procedures and also: If there is electricity, write a letter from the General Manager to all guests and have it delivered to all occupied rooms and posted at the Front Desk Have staff come in early and plan to stay late, especially Night Audit, Front Office Supervisory and Housekeeping Supervisory personnel Transcribe beginning balances from Contingency Reports and all room charges to hard copy guest folios If you use pre-printed folio forms normally, use those for contingency operations. If not, create in advance and maintain a supply of folio forms with key fields (name, address, room, rate, dates, billing, group, clerk, etc) identified in Crash Kits Include Rate, Group, Billing Information and Address from the Contingency Reports onto the folio to facilitate checkouts File by room number in a bucket Prepare hard copy Registration Cards for expected arrivals Create a Rack Sheet from the Contingency Reports and control sheets used during brief outage contingency operations If there is electricity, do it in an Excel worksheet to facilitate changing statuses as time goes on One row per room, with Status, Guest Name, Arrival/ Departure Dates Maintain constant communications with Housekeeping on room status Reconcile Room Availability with Housekeeping at least twice daily during the outage, morning and afternoon Prepare an updated Telephone List with arrivals and departures as necessary and give to operators Prepare for a manual Night Audit 2009 American Hotel & Lodging Association. All rights reserved. 21

30 In practice, most operators will forgo Night Audit reporting during a system failure and simply post room and tax manually to the paper folios established for each room Manually create Housekeeping Assignment sheets during the Night Audit so they are ready when the housekeepers arrive in the morning As checkouts occur, settle the balances manually and record each transaction on a separate control sheet for each cashier that will become their cashier s report Each cashier should advise the rack clerk of what rooms have checked out periodically The rack clerk should update Housekeeping and PBX as checkouts occur Advise guests that the system is down and that a copy of the folio will be sent to them after the system problem is resolved Be certain to capture addresses, or other, in order to honor this commitment to the guest Expect that some guests will demand a copy of their folio and make photocopies if possible Each cashier needs to balance their cash and credit card receipts to their cashier s report at the end of their shift Drop cash receipts as per the hotel s established practices Long-Term PMS Outage (Two or More Night Audits) If it appears that the outage will go on for more than one night audit cycle, then implement the above procedures and also: Establish a room revenue control sheet in Excel to capture room revenue as it is posted by hand to the hard copy folios Use this sheet, plus the room charge control sheets to generate a basic revenue report Secure current availability and expected arrivals reports from the CRO if integrated Store all control sheets (Arrival/Departure; Posting; Cashier Control; Room Revenue) in an accordion file by day to facilitate recovery processes PMS Recovery Processes and Testing The essence of recovering from an extended PMS outage is to bring the system current for each day, run a night audit in the system and to repeat American Hotel & Lodging Association. All rights reserved.

31 as necessary. This process will usually create discrepancies between what was manually posted and collected and what the system says later. Interfaces constitute a major source of these discrepancies: many interfaced systems, especially call accounting systems, will buffer charges until they sense the PMS is there. Then they will dump all of the charges into the PMS, even though the guest in the system on the PMS may have checked out several days earlier. Most hotels will elect to adjust off unverifiable revenue of this sort and move on. Substantial discrepancies need to be resolved, and may require contacting the guest to explain why the charge posted to the credit card is not the same as the amount cited at checkout. If the system was restored from a back-up tape, then the first step in recovery is to bring the system current with the Contingency Reports used when contingency operations first began. Then utilize the chronological control sheets for Arrivals and Departures, Cashiering and Posting to bring the system current day by day. Maintaining those control sheets in an orderly manner is what makes recovery possible. Watch for duplicate or erroneous postings created by interface buffer postings, as these will not be shown on the Posting control sheets. These processes represent an often-overlooked area in Contingency Planning. If a system has been down for a period of time, then when it comes back up, the hotel needs defined processes for getting the data in the system current with the actual state of the house based on transactions performed manually while the system was down. In a brief outage, the system will probably come back up with data current as of the moment it went down, so only manual transactions conducted during the failure need to be re-input. Because the contingency plan processes included steps for tracking all of these transactions, this activity is relatively straightforward. In an extended outage, the system may be brought up with only the data as of the previous night s data back-up or even older. This scenario means that all transactions that occurred between the back-up and the implementation of the Contingency Plan need to be recovered from somewhere (typically other systems such as the POS and IRE system and the Contingency Reports) Testing If the hotel does not test and practice their Contingency Plan, they won t know if it works or how to use it. A planned shutdown is an ideal time to exercise the Contingency Plan American Hotel & Lodging Association. All rights reserved. 23