UNIVERSITY OF CALIFORNIA DAVIS INTERNAL AUDIT SERVICES. Information and Educational Technology Data Center and Client Services Project #08-21

Transcription

1 DAVIS INTERNAL AUDIT SERVICES Information and Educational Technology Data Center and Client Services Project #08-21 June 2009 Fieldwork Performed by: Tony Firpo, Principal Auditor Reviewed by: Leslyn Kraus, Associate Director Approved by: Richard Catalano, Director

2 Information and Educational Technology Data Center and Client Services Internal Audit Services Project #08-21 MANAGEMENT SUMMARY Internal Audit Services (IAS) has conducted a review of the campus Data Center and Client Services (Data Center) organization. The Data Center is a division of Information and Educational Technology and is comprised of four units: (1) Data Center Infrastructure Services; (2) Database and Systems Management; (3) Infrastructure Systems Management; and (4) Operations. The scope of this audit was limited to a review of the Data Center s disaster recovery and business continuity planning, and a review of the Data Center s Service Level Agreements (SLAs) with its clients. To conduct the review, IAS reviewed documentation including the Data Center disaster recovery plan and a sample of SLAs; and interviewed Data Center and campus personnel. Fieldwork on this audit was performed intermittently between July 2006 and March 2009 because of interruptions involving (unrelated) IAS investigations. The existing disaster recovery plan for the Data Center was developed in 2003 based on a business impact analysis completed in Since the impact analysis was completed, significant changes have occurred to computing resources provided by the Data Center and required by the campus to meet operational needs. Accordingly, the disaster recovery plan must be updated to ensure there is an effective and systematic response to an event significantly impacting computing services provided by the Data Center. The plan should be updated only after a risk assessment and business impact analysis has been conducted to identify critical functions and impact of loss or disruption to those functions. The proposed risk assessment and business impact analysis is an essential step in providing information to the Chancellor, Provost, Vice Provost Information and Educational Technology and other senior leaders in weighing various options and updating the disaster recovery plan to properly reduce risk to what is determined to be an acceptable level. It will be imperative that the business impact analysis be a collaborative effort of the Data Center and major clients. Accordingly, campus-wide support for the business impact analysis through the formation of committees to lead the project and/or identification of funding for external consultants will be critical to its success. Dedicated client participation will be critical to thoroughly determine the impact to operations if computer resources are not available for various time intervals (i.e., a day, a week, two weeks, a month). In line with efforts to address the disaster recovery plan for the Data Center, there is a need to effectively communicate and coordinate with clients on the respective roles and responsibilities if a disaster should disrupt services covered in existing SLAs. Consistent with the proposed broader impact analysis, clients who are relying on the Data Center for operating and managing computer resources through SLAs need to evaluate their disaster preparedness. i

3 Information and Educational Technology Data Center and Client Services Internal Audit Services Project #08-21 TABLE OF CONTENTS MANAGEMENT SUMMARY... i TABLE OF CONTENTS... ii I. BACKGROUND... 1 II. PURPOSE AND SCOPE... 1 III. CONCLUSION... 2 III. OBSERVATIONS, RECOMMENDATIONS, AND MANAGEMENT CORRECTIVE ACTIONS A. DISASTER RECOVERY AND BUSINESS CONTINUITY. 2 B. SERVICE LEVEL AGREEMENTS.. 4 ii

4 I. BACKGROUND The campus Data Center and Client Services (Data Center) organization is a division of Information and Educational Technology (IET) and is comprised of four units: (1) Data Center Infrastructure Services, which is responsible for providing both the physical infrastructure for the Data Center and services in support of all systems in the Data Center; (2) Database and Systems Management, which provides database expertise to support many of the campus applications critical to the mission of the University; (3) Infrastructure Systems Management, which provides the infrastructure, computer security and client /business system expertise to support many of the campus applications critical to the mission of the University; and (4) Operations, which provides operational support for all services that are housed in the Data Center. The mission of the Data Center is to provide consistent and secure access to the University s mission critical computing services, and to provide the campus community with excellent computer support and consultation services. The Data Center has a staff of approximately 43 administrative personnel and had a current year budget for fiscal year of approximately $4.6 million, including core funds of $3.3 million and recharges of $1.3 million. II. PURPOSE AND SCOPE This audit was originally planned as a full-scope general controls review, however, in 2008 the Data Center engaged an external consulting firm to perform a comprehensive review of its operations as it related to security. IAS obtained a copy of the consultants report and concluded that because of the work performed by the consultants, security /access controls could be excluded from the scope of our review. Additionally, based on discussions with Data Center and campus management, physical and environmental controls were also excluded from the scope of the review, as management is already aware of certain deficiencies with the Data Center site and is working toward short-term and long-term solutions. As a result, the scope of this IAS project was limited to a review of the Data Center s disaster recovery and business continuity planning, and a review of the Data Center s Service Level Agreements (SLAs) with its clients. To conduct the review, IAS reviewed documentation including the consultants report, the Data Center disaster recovery plan, and a sample of SLAs; and interviewed Data Center and campus personnel. Fieldwork on this audit was performed intermittently between July 2006 and March 2009 because of interruptions involving (unrelated) IAS investigations. 1

5 III. CONCLUSION The existing disaster recovery plan for the Data Center was developed in 2003 based on a business impact analysis completed in Since the impact analysis was completed, significant changes have occurred to computing resources provided by the Data Center and required by the campus to meet operational needs. Accordingly, the disaster recovery plan must be updated to ensure there is an effective and systematic response to an event significantly impacting computing services provided by the Data Center. The plan should be updated only after a risk assessment and business impact analysis has been conducted to identify critical functions and impact of loss or disruption to those functions. The proposed risk assessment and business impact analysis will also be significant as the Chancellor, Provost, and other senior UCD leaders weigh various options to reduce risks to what is determined to be an acceptable level. It will be imperative that the business impact analysis be a collaborative effort of the Data Center and major clients. Dedicated client participation will be needed to thoroughly determine the impact to operations if computer resources are not available for various time intervals (i.e., a day, a week, two weeks, a month). The updated plan needs to include periodic testing. In line with efforts to address the disaster recovery plan for the Data Center is the need to effectively communicate and coordinate with clients on the respective roles and responsibilities if a disaster should disrupt services covered in existing SLAs. Consistent with the proposed broader impact analysis, clients who are relying on the Data Center for operating and managing computer resources through SLAs need to evaluate their disaster preparedness. Past communication has been provided to senior management on the need for better guidance and templates to assist departments with this important task. IV. OBSERVATIONS, RECOMMENDATIONS, AND MANAGEMENT CORRECTIVE ACTIONS A. Disaster Recovery and Business Continuity 1. The disaster recovery plan for the Data Center needs to be updated based on a current business impact analysis. The existing disaster recovery plan was prepared in July 2003 based on a business impact analysis performed in Accordingly, plan assumptions and strategies have not been reassessed along with risks and changes in critical computing needs. Over the past fourteen years since the impact analysis was completed and five years since the plan was developed, campus computing resources have significantly expanded, as has the reliance on them to operate the institution. Certain assumptions made in the current plan may no longer be valid. For example, the plan does not provide for a hot or cold site that would allow for alternative computer support in the event of a major loss to the Data Center. The plan states that alternate sites were not considered because of funding limitations. If an updated 2

6 business impact analysis were performed, management would reevaluate this funding decision, especially now with the collaboration occurring with the Health System. The business impact analysis will require collaboration between the Data Center and its major clients. For instance, impact analyses conducted by Data Center clients will be necessary in order to inform the Data Center s overall impact analysis and disaster recovery plan. University of California Business and Finance Bulletin (BFB) IS-12, Continuity Planning and Disaster Recovery, states that disaster recovery plans should be updated to reflect changing environments, processes, technology, or other impacts as appropriate. The failure to have an updated plan document based on a current business impact study could potentially impair the Data Center s effective response to an emergency situation. Recommendations The Data Center disaster recovery plan should be reviewed and updated as considered necessary based on the current university environment and an updated business impact analysis. Also, procedures should be established by the Data Center for periodic updating of the plan as outlined in the plan document. Management Corrective Action By October 31, 2009 a plan will be developed for conducting the business impact analysis and updating the disaster recovery plan. The plan will address the need for the formation of committees and /or external consultants to lead the project, and will include provisions for inclusion of client risk analysis and business impact analysis. The plan will also include an anticipated timeline for completing analysis work, as well as a timeline for revising the Data Center s disaster recovery plan. The plan will be approved by the Vice Provost Information and Educational Technology and the Provost and will be shared with the Chancellor and other senior leadership groups such as the Council of Vice Chancellors and Deans. 2. The Data Center has not fully tested all aspects of the current disaster recovery plan or developed a schedule for conducting periodic exercises of its disaster recovery and business continuity capabilities. Conducting periodic exercises of disaster recovery and business continuity readiness is considered to be a best practice in the information technology environment. BFB IS-12 states that disaster recovery plans should be tested on a periodic basis by various means, such as disaster recovery exercises, testing of alternate sites, or other simulations of potentially predictable 3

7 emergencies. The Data Center has in the past completed certain disaster recovery exercises; however, these exercises have not been comprehensive or performed consistently in adherence to a regularly established schedule. Recommendation The Data Center should establish a schedule for conducting periodic exercises of its disaster recovery /business continuity capabilities. Management Corrective Action By October 1, 2009 management will establish a schedule for conducting periodic exercises of its disaster recovery /business continuity capabilities. In addition, procedures will be developed to document the results of tests performed. B. Service Level Agreements The Service Level Agreements (SLAs) between the Data Center and its clients do not adequately delineate responsibility for disaster recovery and business continuity planning. The Data Center has executed 45 Service Level Agreements (SLAs), which represent a contractual arrangement between the Data Center and a client outlining services to be performed, responsibilities of each party, fees, etc. Below are examples of clients and projects /services that have computer hardware supported by the Data Center through SLAs: Client Mondavi Center Office of Administration Office of Administration Office of Resource Management & Planning Undergraduate Admissions Graduate Studies Project/Service Paciolan Financial Information System Effort Reporting Databases /Cold Fusion E-Recruitments Reporting IAS reviewed a sample of SLAs for fiscal year 2008 and concluded that the contractual language needed to be improved to effectively communicate the responsibility for disaster recovery and business continuity planning. Specifically, disaster recovery and business continuity is a shared responsibility between the Data Center and its clients, and the SLA language should be strengthened to reflect this fact. For example, currently under the Client Responsibilities section of the SLA, certain elements of disaster planning including required maintenance contracts with vendors and escalation procedures are addressed; however, there is no section that specifically addresses disaster recovery per se. Also, the SLAs do not reflect the fact that Data Center clients running mission critical systems would receive priority over other Data Center clients in the event of a 4

8 large-scale disaster. Lastly, IAS found that the Data Center did not have any established policy concerning the level of client management expected to approve SLAs. Because of the significance of SLAs, especially on critical systems such as DaFIS and the Student Information System, senior management should approve SLAs. Recommendations The Data Center, in consultation with its clients, should develop expanded language within the SLAs surrounding responsibility for disaster recovery and business continuity planning. Additionally, the Data Center should inform its clients of the changes to the SLAs in a memo or letter. Also, the Data Center should require that SLAs for department systems be approved by the respective chair or department head, and for campus wide administrative systems the approval level should be at least at the Assistant Vice Chancellor level. Management Corrective Action By October 1, 2009 management will revise the SLAs to include a section addressing disaster recovery, such that by 2010 SLAs will include the updates. Along with the SLAs, a cover letter will be sent to each appropriate level of client management highlighting the disaster recovery additions. The Data Center also has developed a policy that SLAs for department systems be approved by the respective chair or department head and for campus wide administrative systems the approval be at least at the Assistant Vice Chancellor level. * * * 5