911 Data Center Operations Performance Audit

Transcription

1 911 Data Center Operations Performance Audit June 2010 Office of the Auditor Audit Services Division City and County of Denver Dennis J. Gallagher Auditor

2 The Auditor of the City and County of Denver is independently elected by the citizens of Denver. He is responsible for examining and evaluating the operations of City agencies for the purpose of ensuring the proper and efficient use of City resources and providing other audit services and information to City Council, the Mayor and the public to improve all aspects of Denver s government. He also chairs the City s Audit Committee and oversees the City s Comprehensive Annual Financial Report (CAFR) The Audit Committee is chaired by the Auditor and consists of seven members. The Audit Committee assists the Auditor in his oversight responsibilities of the integrity of the City s finances and operations, including the integrity of the City s financial statements. The Audit Committee is structured in a manner that ensures the independent oversight of City operations, thereby enhancing citizen confidence and avoiding any appearance of a conflict of interest. Audit Committee Dennis Gallagher, Chair Maurice Goodgaine Jeffrey Hart Timothy O Brien Robert Bishop Robert Haddock Bonney Lopez Audit Staff Audrey Donovan, Deputy Director, CIA Stephen E. Coury, IT Audit Supervisor, CISA Robert Pierce, Lead IT Auditor, CISA Aaron Pratt, Senior IT Auditor, CISA Brandon Blomquist, Staff IT Auditor You can obtain free copies of this report by contacting us at: Office of the Auditor 201 W. Colfax Avenue, Dept. 705 Denver CO, (720) Fax (720) Or view an electronic copy by visiting our website at:

3 City and County of Denver Dennis J. Gallagher Auditor 201 West Colfax Ave., Dept. 705 Denver, Colorado FAX June 17, 2010 Ms. Molly Rauzi, Chief Information Officer Technology Services City and County of Denver Mr. Alvin J. LaCabe, Jr. Manager of Safety City and County of Denver Dear Ms. Rauzi and Mr. LaCabe: Attached is the Auditor s Office Audit Services Division s report of their audit of the 911 Data Center Operations for the period January 1, 2009 through January 31, The purpose of the audit was to assess the efficiency and effectiveness of controls related to operating the 911 Data Center, such as managing software changes, patching systems, and providing disaster recovery capability. The audit revealed that while many advances have been made at the data center, procedural improvements are needed to maintain system reliability. If you have any questions, please call Kip Memmott, Director of Audit Services, at Sincerely, Dennis J. Gallagher Auditor DJG/ap cc: Honorable John Hickenlooper, Mayor Honorable Members of City Council Members of Audit Committee Ms. Roxane White, Chief of Staff Mr. Claude Pumilia, Chief Financial Officer Mr. David T. Roberts, Chief Services Officer Mr. David Fine, City Attorney Mr. L. Michael Henry, Staff Director, Board of Ethics Ms. Lauri Dannemiller, City Council Executive Staff Director Ms. Beth Machann, Controller Mr. Mel Thompson, Deputy Manager of Safety To promote open, accountable, efficient and effective government by performing impartial reviews and other audit services that provide objective and useful information to improve decision making by management and the people. We will monitor and report on recommendations and progress towards their implementation.

4 City and County of Denver Dennis J. Gallagher Auditor 201 West Colfax Ave., Dept. 705 Denver, Colorado FAX AUDITOR S REPORT We have completed an audit of the 911 Data Center Operations for the period January 1, 2009 through January 31, The purpose of the audit was to assess the efficiency and effectiveness of controls related to operating the 911 Data Center, such as managing software changes, patching systems, and providing disaster recovery capability. This performance audit is authorized pursuant to the City and County of Denver Charter, Article V, Part 2, Section 1, General Powers and Duties of Auditor, and was conducted in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The audit revealed that while many advances have been made at the data center, procedural improvements are needed to maintain system reliability. Specifically audit work determined that internal controls for workstation patching, antivirus updates, offsite storage of archive backups, documentation of change management procedures, and periodic review of building and system access all need to be improved. We extend our appreciation to the personnel who assisted and cooperated with us during the audit. Audit Services Division Kip Memmott, MA, CGAP, CICA Director of Audit Services To promote open, accountable, efficient and effective government by performing impartial reviews and other audit services that provide objective and useful information to improve decision making by management and the people. We will monitor and report on recommendations and progress towards their implementation.

5 TABLE OF CONTENTS EXECUTIVE SUMMARY 1 Procedural Improvements Needed to Maintain System Reliability 1 INTRODUCTION & BACKGROUND 2 SCOPE 3 OBJECTIVES 3 METHODOLOGY 4 FINDING 1 5 Workstation Patching and Antivirus Updates Not Performed or Monitored for Successful Installation 5 FINDING 2 5 Archive Backups Not Stored Offsite 5 FINDING 3 6 Informal Change Management Procedures 6 FINDING 4 6 No Formal Procedure or Periodic Review for Building and System Access 6 AGENCY RESPONSE 8

6

7 EXECUTIVE SUMMARY Procedural Improvements Needed to Maintain System Reliability Over the past two years significant improvements have been made for the 911 Communications Center and its data center operations. A new Recovery Operations Center has been established which supports complete offsite recovery of both the 911 call taking and the data center in the event of a disaster at the main facility. Control over the installation of software changes have been enhanced, along with many building infrastructure improvements. Although many advances have been made at the data center, formalization and improvements to strengthen existing controls are needed. For example, audit work revealed that the installation of critical security patches and antivirus updates were not monitored to ensure that all required updates were applied. Audit work also found that important data archives were not stored offsite, that production software change management procedures were informal and do not produce evidence of necessary approvals, and that access lists were not periodically reviewed for who has access to the physical building or to critical computer systems. These issues could affect overall system reliability, inhibit the ability to recover important City data after a disaster, or allow the interference or disruption of critical operations. P a g e 1 Office of the Auditor

8 INTRODUCTION & BACKGROUND Advances at the 911 Communications Center The Denver 911 Communications Center serves as the Public Safety Answering Point (PSAP) for 911 telephone calls into the City and County of Denver. Personnel at this center dispatch police, fire, and medical personnel (ambulances) in response to citizen calls for emergency assistance. Over the past two years significant improvements have been made for the 911 Communications Center and its data center operations. Specifically, a new Recovery Operations Center (ROC) has been established which supports complete offsite recovery of both the 911 call taking and the data center in the event of a disaster at the main facility. Controls over the installation of software changes have been enhanced, including the separation of the test, quality assurance, and production environments. Building improvements include enhancements to ventilation, physical access security, and cabling infrastructure. Funding to establish the Recovery Operations Center came from the federal government for the 2008 Democratic National Convention held in Denver. Federal safety and security requirements mandated that the convention have its own dedicated PSAP to serve the area immediately surrounding the convention site. As such, the City received federal grant monies for the purchase and upgrade of hardware and software and staff training. The grant allowed the City and County of Denver to retain all of the physical improvements funded by the convention, thus providing the City with an ongoing recovery capability as critical servers and call taker workstations are backed up at the ROC through duplicate hardware and software configurations. The systems at the ROC are routinely tested to ensure they remain operable and current. Challenges to Keeping Software Up to Date The establishment of the ROC has allowed the City to better address challenges related to software updates. For example, a regular use of the ROC is to host 911 operations while system upgrades are periodically performed on the Computer Aided Dispatch (CAD) system. While the main systems are taken down for maintenance and upgrades, the 911 operations center staff operates from the ROC. This process provides for a controlled and uninterrupted transfer of operations with minimal to no impact on the City s ability to answer 911 calls. In addition to CAD software updates, servers and workstations undergo regular maintenance, patching, and updating. In order to minimize the impact on critical operations, special consideration must be given to both the testing of updates and the time of day for when updates are applied. The ROC provides the City with additional flexibility for this purpose. City and County of Denver P a g e 2

9 Although a rare occurrence, both operating system software and antivirus vendors have released defective updates that have caused system outages to their respective customer bases. 1 Before operating system software updates are applied to the CAD servers, they must be tested and approved by the vendor company that supports the Computer Aided Dispatch System. Before antivirus updates are applied, they must be proven to be stable. The timing for when updates are applied needs to be coordinated so that critical workstations and servers are not re-booted while being used during a production shift. SCOPE The audit examined and assessed the efficiency and effectiveness of controls over data center operations for the 911 Communications Center. The audit period extended from January 1, 2009 through January 31, OBJECTIVES Audit objectives were to ensure: Change controls provided for: the separation of processing environments for test, quality assurance, and production; the separation of duties for the roles performed by software developers, system testers, and end users; and that changes are authorized, tested and approved before being implemented into production; Security settings limited access to authorized individuals for Computer Aided Dispatch (CAD) systems at the application, database, operating system, and physical security levels; Access management controls limited employee access to specific job functions and that access to City systems and data is removed when individuals terminate their employment with the City; Operational controls provided for system backup and recovery capability for the CAD systems; All relevant security patches were installed on all 911 computers; and Antivirus definitions were up to date on all 911 computers. 1 For example, on April 21, 2010, many PCs within the City were not usable due to a defective antivirus update file. P a g e 3 Office of the Auditor

10 METHODOLOGY We utilized multiple methodologies to achieve audit objectives. gathering and analysis techniques included, but were not limited to: These evidence Interviewing personnel in Technology Services and reviewing selected policies and procedures related to CAD and its infrastructure; Utilizing Computer Assisted Auditing Techniques (CAATs) to compare 10,204 employees terminated since 2005 to the population of 1,510 individuals with active user accounts within the CAD system; Directly observing physical access controls in place at both the main and recovery data centers and verifying that individual access to the data center facilities was granted to current authorized employees; Directly observing environmental controls in place at the data centers supporting the CAD systems through onsite inspection and examination of maintenance records; Examining evidence for backup and offsite storage of media; Obtaining access to Active Directory Users and Computers (ADUC) for examining login account access and information; Reviewing Windows Server Update Services (WSUS) reports for security patch status; Reviewing McAfee epolicy Orchestrator reports for antivirus updates; and Reviewing change management processes and procedures for CAD software modifications. City and County of Denver P a g e 4

11 FINDING 1 Workstation Patching and Antivirus Updates Not Performed or Monitored for Successful Installation Technology Services utilizes automated software tools to apply critical system patches and antivirus software updates to City computers. Our testing identified computers with missing updates and others that had not been updated for several years. For the computers missing updates, we found that the software tools did not accurately report their update status and that there was no management follow-up process to ensure that all patches and antivirus updates were being applied successfully. For the computers that were not updated for several years, responsibility for who was to perform the updates had not been established. Workstations that are not patched against known system vulnerabilities and/or do not have up to date antivirus software could be susceptible to malicious computer software that may disrupt normal operations and facilitate unauthorized access and the subsequent disclosure, misuse and/or destruction of sensitive City information. Recommendation 1. Technology Services should establish responsibility for applying tested and approved security patches and stable antivirus updates for all computers at the 911 center and implement a formal follow-up process to ensure the updates are being applied successfully. The timing of updates to critical servers and workstations should be performed during scheduled maintenance periods as to not interfere with critical production shifts. FINDING 2 Archive Backups Not Stored Offsite Two important data archives are backed up to enable the recovery of important historical information related to 911 calls. One data archive is backed up on a real time basis to optical media and the other is backed up on a daily basis to tape media. However, due to incomplete procedures, neither of these two data archives have backups stored offsite. Without offsite storage of backup media, there is an increased risk that important historical data will be lost in the event of a data center disaster. Recommendation 2. Technology Services should develop formal procedures to store important data archive backups at an offsite location, such as at the Recovery Operations Center through the physical transport of media or through remote backup technology. P a g e 5 Office of the Auditor

12 FINDING 3 Informal Change Management Procedures Change management procedures for moving proposed changes into production are not documented and do not provide formal evidence of approvals. Currently, proposed changes to production software are tested from both a systems and end-user perspective. System testing takes place in a development environment and acceptance testing by end-users is completed in a training environment. When both systems personnel and end-users agree to implement the proposed change, the vendor is allowed to install the change on the production server. Although this process provides for separation of testing environments and separation of testing roles, the overall process is not documented and approvals are provided on a verbal basis rather than being formally documented. Without a formally documented and monitored change management process there is an increased risk that unauthorized changes may go undetected which could lead to unintended application downtime or processing errors. Recommendation 3. Technology Services should formally document its production software change management policies and procedures for the 911 center, including its separation of testing environments and separation of testing duties. The procedures should also provide formal evidence of authorization, testing results, and approvals, including user sign-offs. FINDING 4 No Formal Procedure or Periodic Review for Building and System Access The 911 Communications Center was undergoing a major upgrade to its building security access system during the audit that corrected several discrepancies with the former system. Audit work confirmed that only current and authorized employees have access under the new building access system. In addition to building security, we reviewed system user access and identified individuals with inappropriate or unnecessary access. Discrepancies for both building and computer system access are a result of inconsistent procedures, a lack of clear authority for granting access, and the absence of a periodic review of access lists. Without standardized procedures, there is an increased risk that access may not be consistent with employee job functions which may result in employees or former City and County of Denver P a g e 6

13 employees retaining access for which they are no longer authorized. The use of unauthorized access could be used to interfere with or disrupt critical operations. Recommendation 4. Technology Services and Communications Center Management should formalize building and system access procedures to ensure that access is authorized and granted according to employee job function, adjusted when employee roles change, and removed when an employee transfers out of the Communications Center or terminates employment with the City. Procedures should ensure that IDs are unique in order to maintain accountability for both individual building and system access. Both building and system access should be periodically reviewed, perhaps on a quarterly basis, to ensure they remain accurate. P a g e 7 Office of the Auditor

14 AGENCY RESPONSE City and County of Denver P a g e 8

15 P a g e 9 Office of the Auditor

16 City and County of Denver P a g e 10

17 P a g e 11 Office of the Auditor