City of New Orleans. Records Retention Final Report. April 15, 2010

Transcription

1 City of New Orleans Records Retention Final Report April 15, 2010 Prepared for: City of New Orleans Office of Technology M. Harrison Boyd 1300 Perdido St. New Orleans, LA Submitted by: SunBlock Systems, Inc. David Sun 1616 Anderson Rd. #350 McLean, VA 22102

2 1 Executive Summary Search LTC Report IT Operational Issues Search Sources Identified Server based systems Desktop and Laptop computers BlackBerry hand held devices Backup tapes Data Preservation Methodology Server based systems Desktop and Laptop computers BlackBerry hand held devices Backup tapes Search Results BlackBerry Devices Desktop and Laptop Computers Servers Alternative Search Methodology Search Findings Review of LTC Report Operational Issues Appendix SunBlock Systems Records Retention Page 2 of 27

3 1 Executive Summary SunBlock Systems was retained by the City of New Orleans to search the information technology (IT) infrastructure of the Mayor of New Orleans in order to identify and produce s sent to and from Mayor C. Ray Nagin between July 20, 2008 and December 1, 2008, inclusive. In addition, SunBlock was directed to review the E mail Recovery Project Report issued by the Louisiana Technology Council on July 6th 2009 for irregularities. Lastly, SunBlock was directed to document any IT operational issues encountered as part of our efforts. This report documents our findings Search SunBlock conducted a thorough review of the Mayor of New Orleans s IT infrastructure and numerous systems were identified as potentially storing Mayor Nagin s data. Numerous servers, computers from key personnel, and BlackBerry handhelds were forensically processed using BitFlare, EnCase, and other standard computer forensics tools. A review of these devices and searches for backup tapes produced various s but did not provide a comprehensive source of all messages. An alternative search methodology was employed to locate messages. A reciprocal methodology was utilized that reviewed data from all accounts on the Mayor s Office servers for messages from/to Mayor Nagin. Using the reciprocal methodology, over two million messages were searched. Using a revised date range of July to December , thousands of messages were identified and delivered. Based on the systems in place, it is unlikely that any significant number of additional messages will be found. 1.2 LTC Report SunBlock was tasked to review the findings and report issued by the Louisiana Technology Council (LTC) on July 6 th, entitled E mail Recovery Project Report. Issue 3.1: LTC did not identify alternative sources LTC only identified two servers and Nagin s desktop as potential sources of . BlackBerry handhelds, computers for key staff and file servers containing hundreds of thousands of messages were not examined prior to releasing their findings. In addition, alternative search methodologies that would have yielded results were not employed. Issue 3.2: Standard forensic protocols were not utilized Although the use of data recovery software was documented, proper forensic techniques preventing the contamination of the forensic data were not employed. Our review of the hard drives provided by LTC indicate that it was not until June , approximately seven weeks after work began, that LTC or one of its affiliates utilized a rudimentary tool to create a forensic copy of the MNOMail01 server. Industry standard practice includes taking precautions to prevent modification of computer evidence during an analysis. If a forensic copy of a computer drive is to be made, the copy should SunBlock Systems Records Retention Page 3 of 27

4 be made first and then the analysis conducted, not to conduct an analysis first and then make a forensic copy. Failure to follow the proper procedures can cause the loss or destruction of potentially relevant information and contamination of any forensic examinations. Issue 3.3: Results from RecoverData utility were not properly analyzed LTC provided the results from the RecoverData utility in support of their conclusion that 22GB of data was deleted from the mail server prior to their arrival. SunBlock s testing of the RecoverData utility has found an error in the program which can incorrectly state file sizes. The EDB file in question was extracted using alternative tools such as BitFlare and Windows. A comparison of the digital fingerprints for these extracted files provided an exact match. This corroborates the findings of the IT department that that RecoverData did not accurately see any additional data beyond the 66GB available. 1.3 IT Operational Issues Along with the previously described tasks, SunBlock was also asked to document any operational issues encountered that would be of interest. The goal was to assist in updating policies and procedures for the management and operations of the server platform as well as provide the City with best practices recommendations. Issue 4.1: No uniform retention policy exists During interviews, Office of Technology personnel stated that no policy or Acceptable Use Policy existed. Further research by SunBlock located an policy dated May 13, 2008 on the City site at: However, this policy is not widely known. In order for a policy to be implemented, it must be uniformly disseminated so that employees understand it exists. Issue 4.2: Confusion regarding implementation of policies There is significant confusion among City personnel regarding the implementation of policies and procedures provided for retention. Since server space and backup capacity is limited, an IT directive exists which encourages all users to routinely clear their mailbox. Users are often sent reminders to reduce the storage utilization of their inbox. This directive fails to provide proper instructions on how to satisfy requirements of the retention policy referenced above, potentially leading to confusion among users. Issue 4.3: Backup procedures driven by tape budgets Without a formal policy, the Office of Technology has implemented backup requirements and procedures on an ad hoc basis. Budget considerations and procurement issues have undermined backup practices. Initially, backup data was retained for a 30 day period. As systems grew and data doubled in size, instead of purchasing additional backup capability, backup retention times were reduced by half to two weeks. SunBlock Systems Records Retention Page 4 of 27

5 Issue 4.4: Utilizing system backups for archiving data can be transient in nature. Since traditional system backups only operate on a daily basis, they are not able to keep up with the transient nature of . With nightly system backups, a message that is received, read and deleted on the same day will not be properly backed up. A true archiving solutions was not utilized. Many such solutions exist in the commercial marketplace and can address the needs of public records retention by capturing all incoming and outgoing s to an offsite archive. They allow for secure preservation of s and a user is unable to delete their messages from the archive. These solutions are relatively inexpensive and should be utilized. SunBlock Systems Records Retention Page 5 of 27

6 2 Search In conducting the search for s to and from Mayor C. Ray Nagin, SunBlock began with its standard e Discovery protocol successfully used in similar matters. This protocol begins with an overall review of all systems and operational procedures in place without initially focusing on any specific sources of . This comprehensive approach provides a broad understanding of all potential sources of data, allowing for more reliable identification of all possible sources of responsive s Sources Identified The SunBlock e Discovery protocol began by providing City technicians with a multi page survey consisting of 24 questions regarding systems and operational procedures in place at the Mayor s Office of Technology. The survey was followed up with phone interviews and face to face meetings at the City s Office of Technology, the City Attorney s Office, and Mayor Nagin. Based on the data provided, it became clear that the Office of Technology administers two different systems: one for all City employees, and one dedicated to the Mayor s office and personnel. It was determined that SunBlock would confine its search to systems dedicated to the Mayor s office and personnel. With exclusive focus on the systems from the Mayor s office, numerous sources of data were identified. The sources included server based systems, Desktop and Laptop computers, BlackBerry hand held devices, and backup tapes Server based systems Based on the review, numerous computer servers were identified that could potentially have traces of data. These systems store and/or transfer as part of their operational roles or have had data placed on them in the past as a result of a specific project. These servers include the following machines: Server Function Storage Server Alias Size (GB) Initial Mayor server 136 MNOMail01 Newer Mayor server 686 Mail2 Production BlackBerry 60 Blackberry v Server Retired BlackBerry Server 68 Blackberry p File server 514 File1 Domain Controllers 164 DC1, DC2, DC02p IT Lab Server 137 ITLab Retired SPAM Server 34 SPAM Backup Server 68 Backup Retired File server 68 OldFile Total 1,931 SunBlock Systems Records Retention Page 6 of 27

7 Note: Due to possible public release of this document and related security risks, actual machine names have been replaced with aliases for devices not previously disclosed Desktop and Laptop computers As part of the search, desktop and laptop computers which may have stored s of interest were identified. These devices are of interest since they may store to or from the Mayor either in visible files or previously deleted data available through a forensic review. Based on our interviews with IT personnel, the Mayor, and his staff, it was determined that only the Mayor and his assistant Patricia Smith had access to in the Mayor s account. This security setting was verified by SunBlock during the review. In addition, it was determined that Michael LaFrance from the IT department stored copies of the Mayor and other user s on his computers. This is due to his operational responsibilities supporting backups and processing of public records requests. In all, the following machines were identified and secured: Mayor Nagin desktop Mayor Nagin laptop Patricia Smith desktop Patricia Smith old desktop Michael LaFrance desktop Michael LaFrance laptop Michael LaFrance old desktop In addition to the above, a search for old machines from existing employees or those of departed employees who may have accessed the Mayor s was conducted. The following machines were not available as they are believed to have been repurposed or destroyed. Michael Bevins desktop Wayne Gatlin desktop Mayor Nagin old desktop BlackBerry hand held devices The Mayor is a heavy user of his BlackBerry handheld. As such, his current and previous BlackBerry devices are expected to potentially contain s of interest. During the review, three different devices reportedly used by Nagin were located and secured. Data on these devices are reported to go as far back as November 20, The following BlackBerry handhelds were located: SunBlock Systems Records Retention Page 7 of 27

8 BlackBerry 8830 BlackBerry 9530 BlackBerry Backup tapes A review of the backup procedures produced a lack of pertinent tapes. The operational procedures in place by the Office of Technology only retained data for a two week period. Any backup tapes older than two weeks were overwritten. As such the oldest set of backup tapes retained was made in February In addition, no formal archiving system was in place. Such a short retention period is unusual. This appears to be due to procurement issues and human error. Historical documentation provided by Ciber noted the two week retention period as far back as June In January 2008, problems with purchasing more tapes threatened to reduce the retention period to one week. New tapes arrived in May In addition to procurement issues, as disclosed during an interview with IT personnel, a city contractor in charge of maintaining consistent tape backups for various systems including the server failed to properly perform this duty. The contractor is no longer employed by the City. Offsite storage is currently being provided by Iron Mountain but this procedure was implemented in March 2009 and no backup sets from the period of interest were ever stored there. At SunBlock s request, a procurement records review was conducted by the Office of Technology s Contracting Manager to verify that no other offsite storage vendors are utilized by the IT department for the City. While the review did not identify any known backup sets from the desired time frame, SunBlock s standard protocol includes a physical inspection of the data center and related areas. This inspection is designed to assist IT personnel in potentially identifying data sources which may have been overlooked during previous searches. Given the SunBlock Systems Records Retention Page 8 of 27

9 physical layout of the City s data center, the inspection encompassed many office suites as well as an inspection under the data center flooring. Data Center Inspection Sample Photograph 1 Data Center Inspection Sample Photograph 2 SunBlock Systems Records Retention Page 9 of 27

10 Data Center Inspection Sample Photograph 3 SunBlock Systems Records Retention Page 10 of 27

11 Data Center Inspection Sample Photograph 4 Data Center Inspection Sample Photograph 5 SunBlock Systems Records Retention Page 11 of 27

12 As part of the inspection, additional backup tapes were identified. Three distinct sets of tapes were found. While the contents of these tapes were not cataloged, they are not believed to contain data from the specified period of interest due to the types of tape media found. However, based on the older tape media used and physical labeling, one or more sets may contain data from a period prior to the time frame of the requested. At the request of SunBlock, 44 members of the City Hall MIS staff participated in a search for tapes encompassing their work space, home and other possible storage locations under their control with no further results. Based on this, no other backup tapes are believed to be available. Backup Tapes Found Photograph 1 SunBlock Systems Records Retention Page 12 of 27

13 Backup Tapes Found Photograph Data Preservation Methodology Once data sources were identified, the next phase of the review consisted of properly preserving all potential evidence. Wherever possible, a forensic preservation of the potential sources was performed. Doing this at the earliest possible opportunity is a fundamental component of any forensic effort. It is well known that electronic evidence is highly fragile in nature and that forensic data is lost over time though the normal operation of a computer system. In addition, it is also known that unnecessary modifications of the computer prior to preservation should be avoided as it may spoil potential evidence Server based systems Since some of the servers for the Mayor of New Orleans system were physical devices while others were virtual machines, different techniques were employed for server based systems. For virtual servers, the VMDK files which comprise the virtual server were copied and digital signatures were calculated to verify data integrity. The physical servers lacked USB 2.0 ports and were therefore forensically captured using EnCase Forensic, a common forensic tool Desktop and Laptop computers The desktop and laptop computers were forensically preserved using BitFlare. Due to the desire to remain as cost effective as possible and given time constraints, BitFlare was utilized for the forensic preservation and examination. City IT staff used BitFlare to create secured forensic copies of hard drives as well as perform in situ searches and extraction of forensic data. This was facilitated by BitFlare s automated Chain of SunBlock Systems Records Retention Page 13 of 27

14 Custody and independent validation functionality. In addition, by specifying the appropriate Evidence Discovery Pack, preliminary results were extracted without the need to wait for a laboratory analysis. Fragments of responsive s from the desktop and laptop computers were immediately extracted and available for review BlackBerry hand held devices The three BlackBerry devices were backed up using the RIM Desktop Manager. These data files were then saved for subsequent analysis using Amber BlackBerry Converter Backup tapes Preservation of the tapes consisted of documenting the tapes and securing their location in case future analysis is warranted. The February 2009 MNOMail01 backup tape was secured by SunBlock. The Mail2 tape was verified to be at the Iron Mountain storage facility where any subsequent transfers to City officials would be documented. The backup tapes found during the inspection were secured in a tamper evident box, preserving Chain of Custody pending further examination by SunBlock. Further examination however was hampered when a concurrent law enforcement investigation removed this container of tapes. 2.3 Search Results Based on the review of all systems and operational procedures in place, no single complete and reliable source of s for the Mayor in the time period of interest was identified. This is due in large part to the lack of a formal archival system in place and the short term retention of backup data. However, using a combination of new techniques employed by SunBlock, many partial sources of data were identified and examined. Given the large number of sources identified, limited resources available, and previous work performed the by the IT department and other entities, SunBlock focused its efforts on new areas most likely to generate additional results. A detailed discussion of those sources and the resulting messages follows BlackBerry Devices A review of the three BlackBerry hand held devices reportedly used by Mayor Nagin only provided one device with relevant s. Two of the devices did not contain relevant s. Table 1 below is a table describing the data available for each device. BlackBerry Device s BlackBerry BlackBerry BlackBerry (11/20/2003 8/5/2009) Table 1. Data Available from BlackBerry Handhelds. SunBlock Systems Records Retention Page 14 of 27

15 One significant finding from the review of the BlackBerry devices is that older messages were available on the handheld and not all messages were deleted. These messages did not match Nagin s messages on the server. This would contradict a theory provided by the Office of Technology in their response to the LTC preliminary report. That theory stated that messages were systematically removed from Mayor Nagin s inbox because he deleted messages from his BlackBerry and it had a setting of Handheld wins. If this theory was correct, there would be a higher degree of correlation between the messages on the BlackBerry and those on the server Desktop and Laptop Computers In this search, the desktop and laptop computers were reviewed for data. Outlook data files from Patricia Smith and Mayor Nagin s computers were reviewed. Mayor Nagin s Outlook data files contained only a few s. In an interview, the Mayor stated that he operated under the belief that his messages were being preserved independently of his actions. As such he would routinely delete his s after reading them to keep his inbox clear. Confusion regarding the retention policy may have contributed to this. To further search for data, a forensic examination was also conducted of all relevant desktop and laptop computers. fragments were extracted by BitFlare as part of an in situ forensic analysis and subsequently reviewed. While numerous fragments were recovered, the difficultly in reconstructing these fragments to create entire messages for the time period of interest required directing search efforts to alternative sources. The ability to forensically search these devices remains should sufficient resources be dedicated to this task Servers A preliminary forensic review of the servers was conducted. However, extensive changes were made to the forensic data on the server during the LTC analysis and a subsequent review would have required considerable effort to distinguish between activities from the prior analysis and true operational activity. This, combined with the higher likelihood of success using alternative search methodologies to search unmodified sources, led to a halt of forensically examining the servers. Since the servers were forensically preserved, such an examination may easily be conducted in the future if necessary. Other servers such as file servers and machines used to process public records requests were examined. While working with City IT personnel, various PST files were identified and analyzed. These PST files allowed for multiple search methods to be employed. SunBlock Systems Records Retention Page 15 of 27

16 2.3.4 Alternative Search Methodology Without a single comprehensive repository of s for Mayor Nagin, an alternative search methodology was developed. Instead of simply searching Nagin s inbox for messages, a reciprocal search was implemented in which each potential recipient s mailbox was searched for messages from/to Nagin. Such an approach has limitations as it is highly unlikely to cover the entire universe of s sent/received by Nagin and is subject to the discretion of each recipient s deletion/retention preferences. However, it did generate messages that can be reasonably linked to Nagin s account. Using this reciprocal search methodology, a review of data from all user accounts on the Mayor s Office system was conducted. messages were gathered from the Mayor s Office servers as well as the File Servers since numerous Outlook PST files were identified on the network drive for users. These PSTs are database files created by users to facilitate storage of messages without utilizing server storage. This is often done by a user in response to mailbox storage limits, archival of messages, personal preferences and other operational needs. In all, 153 User PST files were identified from the file server in addition to the messages stored on the servers. As part of this reciprocal search, a mailbox for Seletha Nagin was located. This account was of particular interest as numerous personal messages to or from Mayor Nagin might be expected in this account. While numerous messages did exist in this account, SunBlock s inspection identified a continuity gap in the account. The account was used daily and has numerous messages which span a multi year period. However, from May 7, 2008 to October 8, 2008 the account lacks any messages. Numerous messages appear before and after this time period. In addition, a review of Calendar, Contacts and Notes for the account shows a similar lapse in entries. No explanation is available at this time for this. Given that Mrs. Nagin only accessed her account using a BlackBerry, it is unlikely her actions would have caused this gap. This activity gap is possibly due to a system level change with the account. While it is difficult to know the exact change, possible examples of changes that may explain this gap include: 1) removal and subsequent partial restoration of the account and 2) temporary abandonment of the account with messages being redirected elsewhere. Not withstanding the gaps in messages, the reciprocal search examined over two million messages from the Mayor s Office server were searched. Those messages, combined with data from Nagin s City account provided by IT personnel, resulted in over 20,000 messages to/from Nagin within a revised date range of July 1, 2008 to December 1, Duplicates and calendar entries were removed and ultimately 4,946 unique messages were produced for the period of interest. SunBlock Systems Records Retention Page 16 of 27

17 2.4 Search Findings Based on the search conducted the following conclusions can be made: Finding #1: A large number of systems potentially containing data were examined and thousands of s were produced. Using various techniques including interviews of key personnel, physical inspections of City Hall IT facilities, and forensic techniques, numerous sources of data were identified. Sources omitted by previous efforts such as file servers and computers were reviewed. Additional tapes and potential sources of data were discovered. Sources deemed most likely to be fruitful were examined and thousands of messages were identified. Finding #2: An archive of Mayor Nagin s s was not created or maintained. No single source of Mayor Nagin s was identified. Backup tapes from the relevant time period were not retained. No formal archive system was implemented. Finding #3: It is unlikely that any further searches will yield substantive results. Based on the information gathered and the amount of data examined, it is unlikely that further efforts will yield substantive results. While a forensic review of the hard drives may produce additional message fragments, searching for these fragments based solely on date sent is impractical. Since fragmentation of deleted data often results in the message body being separated from the envelope information containing the date sent, reconstruction of these messages would be cost prohibitive SunBlock Systems Records Retention Page 17 of 27

18 3 Review of LTC Report In addition to the search, SunBlock was retained to review the report and findings from the July 6 th Louisiana Technology Council (LTC) entitled E mail Recovery Project Report for irregularities. Several issues were identified with methodology and findings detailed in the report. The issues were determined based on a review of the report, a preliminary examination of the hard drives provided by LTC from their analysis, analysis of tools employed by LTC, discussions with personnel at the Office of Technology, and a review of the Office of Technology Response to State Irregularities document. Due to pending litigation, discussions with personnel from LTC were not possible. SunBlock reserves the right to modify these findings pending additional information or discussions with LTC personnel. Issue 3.1: LTC did not identify alternative sources or search techniques From the LTC report, only two servers were identified to be of interest. Tape restorations of those servers (primarily MNOMail01) were performed but as stated in the report, LTC s work was confined to the City s two mail servers. LTC also states that the two most likely places for recovery of all the messages were on the Mayor s desktop computer in the form of OST files. While the Mayor s desktop computer is an additional source of messages, it is clear that there exist numerous other sources of data not identified by LTC. As was demonstrated by the Search, other sources of data included the BlackBerry handhelds, computers for key staff such as his assistant, Mayor File and servers as well as possibly the additional backup tapes that were found through a physical review of the site. Many of these additional sources would have been determined by conducting a thorough review of the IT system as well as asking personnel about additional backup tapes and inspecting the facilities. In addition, alternative search methodologies that would have yielded results were not employed. A reciprocal search methodology produced thousands of relevant s. Issue 3.2: Standard forensic protocols were not appropriately employed Based on the report provided by LTC, forensic protocols were not appropriately employed. While the use of data recovery software was documented, proper forensic techniques preventing the contamination of forensic data were not employed. Considerable forensic data was modified as part of the LTC analysis. User accounts were created, programs were installed and data was restored on the servers themselves. All of these activities caused potentially precious forensic data to be destroyed. While traces of the information still exist, the modification undermines its usefulness and increases the expense required to examine the data. SunBlock Systems Records Retention Page 18 of 27

19 It should be noted that, while not documented in the LTC report and based on forensic data examined, LTC or one of its affiliates finally did utilize the rudimentary DD utility to create a forensic copy of the MNOMail01 server on June 21, 2009 at 4:50PM. DD is a system administration tool which can also serve as a copying program to make forensic copies of a computer hard drive. A forensic copy of the MNOMail01 data was found on drives provided by LTC from their analysis. This forensic copy was dated June 22, 2009, approximately seven weeks after work began and only days before LTC halted their work. By this time, LTC had already modified data on the MNOMail01 server as user accounts for LTC personnel and some of the various data recovery programs utilized by LTC were contained within that forensic image. Industry standard practice includes precautions to prevent modification of original computer evidence. Forensic copies of a computer drive, if made, should be created before the analysis is conducted, not afterwards. Following the proper procedures can prevent the loss or destruction of potentially relevant information and contamination of future forensic examinations. Issue 3.3: Results from RecoverData utility were not properly analyzed In the LTC report, much is discussed regarding the City Hall information store and purported deletion of 22GB worth of data. Screenshots provided in their report, repeated below, demonstrate the basis for this assertion. The report states that that the first screen shot from within Microsoft Windows depict the database on MnoMail01now 66Gb[sic] and that the second screenshot from the RecoverData tool indicates what can be seen as available for recovery (meaning that these files were Deleted and may be able to be recovered). The size of the file shown as potentially recoverable is 88GB. From this, LTC states that 22GB was deleted from the Priv1.edb database. The most obvious error in this statement is in reference to the 88GB (Gigabyte) value. An examination of the screenshot from the LTC report clearly shows the size value listed by RecoverData is (88,875,008) which equal approximately 88 Megabytes. A Megabyte is 1/1024 th of a Gigabyte. RecoverData does not specify any units but the screen shots below comparing the size for other files corroborate that RecoverData displays file sizes in the unit of bytes. SunBlock Systems Records Retention Page 19 of 27

20 Unit is bytes As a result of this, there is no evidence to support statements regarding 22GB of missing data. In fact, if one were to follow the same logic applied in the LTC report, using a SunBlock Systems Records Retention Page 20 of 27

21 correct reading of the values provided by RecoverData, one would come to the conclusion that the server contains 87.9GB of additional data. To further examine the accuracy of RecoverData, the size of the Prive1.edb file indicated was compared with the output from other forensic tools. Below is a screen shot showing some of the comparisons. RecoverData displays a file size for the files which are inconsistent with those displayed by other forensic and standard tools. The following table summarizes the comparison of the file sizes between RecoverData and other tools. Tool Priv1.edb Priv1.stm Result Windows (KB) 67,195,656 KB 5,228,552 KB Converted to bytes 68,808,351,744 5,354,037,248 EnCase (bytes) 68,808,351,744 5,354,037,248 BitFlare (bytes) 68,808,351,744 5,354,037,248 RecoverData (bytes) 88,875,008 1,059,069,952 SunBlock Systems Records Retention Page 21 of 27

22 LTC also stated in the Report that these files[priv1.edb and priv1.stm] were Deleted and may be able to be recovered. As can be seen in the prior screenshot, RecoverData displays the Priv1.edb and Priv1.stm files in black. Documentation for RecoverData shown below clearly indicates that deleted files would be listed by in red. The following screenshot created by SunBlock with RecoverData examining the MNOMail01 server verifies that this red/black labeling is in use. The deleted file indicated is one of many log files that a mail server would automatically generate and delete as part of its normal operation. SunBlock Systems Records Retention Page 22 of 27

23 Given this red/black labeling, there is no indication by RecoverData that the priv1.edb and priv1.stm files are in a deleted state. The Office of Technology, in their response to the LTC preliminary report, used RecoverData to extract the Priv1.edb file. They stated that, when extracted by RecoverData, the resulting Priv1.edb file was 67,195,656KB in size. This matches the file size listed by other tools for this file. Their extraction of the Priv1.stm file produced a file 5,228,552 KB in size, which also matches with the size reported by other tools. These results would indicate that RecoverData is referring to the exact same set of data as all of the other forensic tools employed while listing a different, likely incorrect, file size value. LTC appears to have utilized the incorrect results from RecoverData and/or a misinterpretation of the unit of size as their basis that 22GB of data was deleted from the mail server. SunBlock Systems Records Retention Page 23 of 27

24 4 Operational Issues To fully leverage the information gathered as part of this review, SunBlock was also tasked with documenting operational issues identified as part of the search efforts. At this time, SunBlock has identified numerous operational issues relating to the Office of Technology and the handing of electronic public records. In general, there is a lack of policy management pertaining to the handling of electronic public records, specifically s. Without proper policy management, policies are not created, implemented, or enforced in a necessary fashion. In addition, pool policy management can also lead to improper implementation and maintenance of technology solutions. Specific issues identified include: Issue 4.1: No uniform retention policy exists When interviewed, Office of Technology personnel stated that no computer, policy or Acceptable Use policy existed. Upon further research by SunBlock, an policy dated May 13, 2008 was found on the City web site at: However, when discussed with various personnel in the Office of Technology as well as the City Attorney s Office, a range of responses were provided regarding this policy. These responses ranged from not knowing that such a document existed to indicating that the document was published but was not formally implemented. In order for a policy to be implemented, it must be fully disseminated so that employees understand it exists. Procedures which inform employees how to comply with the policy should also be established. Lastly, policies must be enforced to be effective. The current state of awareness would indicate that for all practical purposes, no formal policy for management exists. Without such a policy, it is impossible to properly implement an management solution. It is recommended that the City Attorney s Office work with the Office of Technology and other necessary resources to fully define a public documents retention policy. Issue 4.2: Confusion regarding implementation of policies Without proper policy management, there is significant confusion among City personnel regarding the proper implementation of policies and procedures for retention. After numerous interviews with City personnel in the IT and Law departments, there was obvious confusion regarding the existence of an retention policy. Establishing a uniform policy that is clearly disseminated to all users along with procedures on how to comply with the policy is paramount to compliance. For example, server space and backup capacity is limited so an IT directive exists which encourages all users to routinely clear their mailbox. Users are often sent reminders to reduce the storage utilization of their inbox. This directive fails to provide SunBlock Systems Records Retention Page 24 of 27

25 proper instructions on how to satisfy requirements of the retention policy referenced above. Specifically, the policy states: This retention policy is secondary to City of New Orleans policy on Freedom of Information and Business Record Keeping. Any that contains information in the scope of the Business Record Keeping policy should be treated in that manner. All City of New Orleans information is categorized into four main classifications with retention guidelines: Administrative Correspondence (4 years) Fiscal Correspondence (4 years) General Correspondence (1 year) Ephemeral Correspondence (Retain until read, destroy) While Administrative and Fiscal correspondence retention is managed by IT, the policy also states: The individual employee is responsible for retention of General Correspondence. This lack of uniform policy management can create confusion from the users. One example could be Mayor Nagin s belief that his messages were being preserved independently allowing him to delete his s after reading them. Issue 4.3: Backup procedures driven by tape budgets Lacking a uniformly disseminated and enforced retention policy, the Office of Technology has developed backup requirements and procedures on an ad hoc basis. This has allowed budget considerations and procurement issues to undermine industry best practices for backup and document retention. Specifically, over a period of time, the backup retention window was reduced due to the lack of tapes and budget necessary to purchase more. Prior to July 2006, system backups were available for a 30 day period. From July 2006 to June 2007, systems were expanded and data storage utilization approximately doubled in size. While the purchase of additional backup capability was requested, no action was taken. As a result, backup retention times were reduced by half to two weeks. In general, industry accepted practices involve nightly backups of relevant data. These backups should be retained for a two to four week period. In addition, full monthly backups should be made and retained offsite for a twelve month period. Lastly, annual backups should be retained offsite for the duration specified in a document retention policy. These backup tapes are an important component to the disaster recovery procedures of any IT system. Given the size and complexity of the City s IT system, other technologies such as disaster recovery sites and SAN replication may be incorporated but the ability to restore the entire system to various, predefined points in time must be maintained. SunBlock Systems Records Retention Page 25 of 27

26 Issue 4.4: Utilizing system backups to perform archiving data can be fairly transient in nature. In addition, SPAM and other messages can clutter a user s inbox or BlackBerry, making them unwieldy. Traditional system backups only operate in a daily basis. As such they are not able to keep up with the transient nature of because only the data which exists at the time of backup is preserved. If a message is received, read and deleted on the same day, it will not be preserved during the daily backup. Exceptions to this can be made by utilizing the Exchange dumpster settings but ultimately this is not a practical way to perform archiving. Given the increased oversight due to Sarbanes Oxley act of 2002 in the private sector, many reliable archiving solutions have been created. These solutions are directly applicable to addressing the public records retention problem. These solutions capture all incoming and outgoing s and can store them at an offsite archive. They allow for secure preservation of s and a user is unable to delete their messages from the archive. The solutions can also allow for search and retrieval of messages responsive to any public records request. Prices vary between solutions but one vendor offers a solution for $25/user for 1 year retention and $45/user for 10 years retention. Volume and government discounts may be available. SunBlock Systems Records Retention Page 26 of 27

27 5 Appendix The following software tools were utilized during the course of this review. Amber BlackBerry Converter (ABC Amber) BitFlare (SunBlock Systems) BlackBerry Desktop Manage (Research in Motion) Discovery Attender (Sherpa Software) EnCase Forensic (Guidance Software) Exmerge (Microsoft) Fsum (SlavaSoft) RecoverData (NTFS) Technician Edition (Recover Data) SunBlock Systems Records Retention Page 27 of 27