EMAIL AUDITING, LOGGING AND REPORTING June 2007 INTRODUCTION Corporate Governance, Accountability, Regulatory Compliance, Fraud, Fines, Penalties In the last few years, state and federal legislators and regulatory bodies have implemented a substantial number of regulations designed to force companies to higher levels of accountability and information security standards. Failure to comply with these regulations put businesses at great risk. The consequences of non-compliance are significant and include enforcement actions with fines up to $1M and other penalties. Many of these regulations, such as Sarbanes-Oxley focus on corporate accountability, but have major ramifications on how computer audit logs are handled. Other industry specific guidelines focusing on financial institutions provide very specific tasks relating to the collection, retention and review of logs from email and other corporate systems and applications.
The challenge for IT organizations to provide systems to manage these regulations is exasperated by the fact that they must also: Manage organizational growth (through consolidation and acquisition) which results in highly dispersed and mixed computer and email systems Manage increasing infrastructure costs (throughput, storage, etc) Continue to battle the external onslaught of spam, viruses, malware and other harmful attacks Manage the collection, retention, and auditing of large volumes of good email To effectively address these problems, manage regulations, reduce liability risks and lower costs, organizations should evaluate their email logging and auditing strategies on an on-going basis. With today s ever increasing volume of email communication, the collection, retention and auditing of email must be addressed using an entirely new approach. Logging and Auditing Requirements As the importance of email logs has increased, so has the confusion surrounding the legal issues of this data. On one hand, many believe that email logs must be preserved in a pristine, unalterable format in order to be considered legally valid, while others believe that practical considerations allow a somewhat more flexible standard. Likewise, some professionals may claim that sampling filtering of log records is an acceptable approach, while other evidence suggests that filtered data present major obstacles to admissibility or credibility as evidence. The strategies outlined in this paper draw upon published opinions as well as comparisons to other forms of evidentiary standards to present the argument that: Complete, accurate and verifiable is the criteria that email logs (and other computer logs) are to be held Filtering or sampling of log data is an unacceptable violation of this standard The preservation of log information is critical, not the format of file organization containing that information In addition, the strategies discussed in this paper address various common approaches to log management, along with an examination of the potential legal and technical obstacles that can arise with their implementations. Finally, we conclude by outlining a best-practice approach and solution to auditing, analyzing and reporting on your email communications. Logging Log data is an organization s richest information asset for assessing security posture, tracking sophisticated threats, and meeting audit requirements. Because of their evidentiary value, email logs must be managed as a legal record; they must be complete, accurate and verifiable. Email and computer logs are no longer just trouble shooting tools for techies. They have major legal consequences for any enterprise which uses them. Email logs represent a large portion of business communications. As a result, email log management can make serious demands on a company s technological infrastructure. A large enterprise can generate terabytes of message data in logs alone, pressuring IT administrators, legal counsel and the risk management team to decide what to keep and how to manage the deluge of data. These decisions, often guided by pressures of IT budgets alone, should be informed by the legal ramifications at the onset. The elements common to these legal ramifications are completeness, accuracy and verifiability. Completeness Completeness in the context of logs means two things: individually, that activity is captured without gaps in time and collectively that logs throughout an organization are mainlined in the aggregate. With complete logs one can reconstruct the: who, what, when, where, why and how of an activity involving email logs. The protection of privacy and prevention of theft/misuse of personal information has become more than just a good idea it has become the law of the land
Complete logs enable a digital chain-of-custody which mimics the court-tested method of proving that evidence is original and authentic. A full and complete set of log data provides a truly objective picture of the digital landscape. This makes it possible for investigators, fact finders, and even legal opponents to look at the data and reach the same conclusion. Accuracy For the same types of reasons, accuracy is a prerequisite for the successful use of logs in legal actions or in the context of compliance audits. Corporate due diligence and regulations like Sarbanes-Oxley are meant to ensure the accuracy of financial statements and the underlying IT controls. Accuracy of email log data means that the time, date and content of that log are the same as when it was created. Electronic copies are considered to be best evidence only if they accurately reflect the original. Verifiability If logs are to earn the labels of complete and accurate they must be verified as such. Some techniques such as hashing provide a digital fingerprint of logs that allows verification that log evidence is authentic days, months or years later. Other techniques used to enhance verifiability include the process of documenting each step of the log management process, creating a repeatable digital-chain of custody and storing the data in multiple separate locations. Log Data as Information If ensuring logs are complete, accurate and verifiable is the first step to managing logs, then the second step is to figure out how to turn all of the data into an information resource. It is essential to be able to extract information from the terabytes of log data the enterprise generates--quickly and in compliance with legal standards. Presentation analysis and reporting of logs is critical if they are to be human readable and useful in legal actions. One of the key goals for enterprises is to manage legal risk and avoid legal costs, so self-policing and cooperation with enforcement officers and investigators is important. To do that effectively, logs must be reviewable and understandable. Access to compliance data and significant events, as well as disclosure of the same, requires effective log analytics. Sarbanes-Oxley, for example, requires companies to disclose timely information to the public regarding material changes to the financial condition or operations of the company. Moreover, the Federal Trade Commission maintains in its Safeguards Rules that it is critical to monitor, use, and review access records and logs. Logs as Evidence For CIO s and general counsels, logs become a part of the solution for managing legal risks connected to the control of information. Email logs are increasingly used as audit logs which are the primary evidence to demonstrate the reliability of electronic data and the processes used to create, manage, store and provide digital information. With legally engineered logs, companies can reduce the potential of losing a lawsuit, diminish the costs associated with discovery and defense, and increase the likelihood of forcing an opponent into settlement. Enterprises today should think of email and event logs as critical information and as an asset, not just terabytes of information that you hope you never have to access. And like any asset, they should be managed accordingly: safeguarded against threats and collected and stored in a manner that adds value to company s business by reducing legal risk. Computer-generated logs once a source of data that only the most die-hard techie could embrace have emerged as one of the key chess pieces in legal risk management
Log Management Approaches Many commercial hardware/software vendors and in-house developers have both attempted to meet the challenges of log management by implementing common approaches. Manual Collection and Review Even now, a surprising number of enterprises continue to perform log review of mission critical applications and systems in a decentralized, ad hoc manner. In some such situations, organizations tend to lack a central policy or strategy for regular review of email audit trails and other system logs. In other situations, for example large government organizations, the availability of personnel make scaling through manpower a viable alternative to scaling through software. The potential problems with this approach are numerous: it is error prone, manpower intensive, and provides little or no ability to identify incident or trends by corroborating messages from disparate email and other computer systems. Perhaps the most significant weakness in this approach is that any ad hoc or informal approach to log review will be subject to rigorous scrutiny, and possibly ruled impermissible in a legal challenge. For these and other reasons, most firms with any significant volume of email and other computer traffic have moved to the next approach, in-house development. In-house One of the most common approaches is often built upon in-house developed utilities created for email and computer systems. Common approaches include creating central syslog servers, extensions to log rotate scripts, and command line or web CGI utilities to perform queries against the data and generate reports. Enterprises using this option quickly realize the limitation of working with raw logs. Increasingly voluminous and disparate email systems create management challenges: compressing files to save space lead to substantial decompression penalties, and correlating information from different log types requires complete parsing of records during queries. As a result, historical analysis and investigations become impractical if not impossible to perform in reasonable time periods. Moreover, this approach requires that in-house developers acquire the subject matter expertise to interpret the underlying log files in a meaningful way. From a legal perspective, organizations adopting this option must also contend with the issue of demonstrating the authenticity of log file information. Organizations implementing home-grown systems may also be subject to more arduous proof of the reliability and accuracy of their system compared to organizations adopting commercial products whose reliability has been established. In-house or Commercial Products built on Legacy Systems Another option adopted by both commercial and in-house developer s centers on storing log data in a relational database. Each record is broken out into specific fields stored as columns. This approach often includes a data normalization process, i.e. storing disparate log types into a common schema. This option provides flexibility in constructing queries for log data investigation and analysis as long as the database is properly tuned and indexed for the anticipated queries. The associated overhead with queries which cannot take advantage of an index however can be prohibitively slow or exceed available system resources. Challenges with homegrown systems compressing files to save space lead to substantial decompression penalties, and correlating information from different log types requires complex parsing of records during queries
A number of performance issues arise when using relational databases in a high-volume log management architecture; many of these issues, including insertion and query performance, disk usage and index degradation, are directly related to how the database is indexed. Organizations using RDMBS-based products frequently discover that the solution fails to handle the volumes of logs generated by the enterprise. It also fails to retain the data over sufficient time periods. As a result, log files are often filtered to include only what the vendor considers being events of interest, and data must be purged after relatively short periods. This type of filtering risks omitting valuable information as well as violating the completeness criteria. Because of the issues associated with the use of RDMBS system for long term/high-volume log management, a number of vendors have begun to develop alternative or hybrid systems attempting to provide the flexibility of SQL with storage arrangements that are better suited to log data. These systems suffer many of the same issues associated with relational databases. Log Management Best Practice Approach To effectively address these problems, organizations should look at solutions that are architected to address the specific problems of email audit data collection, retention and analysis. Enterprise-class systems should be designed to provide the high performance, scalability and compression required for large volume log management and compliance needs. To ensure the required information, reports and alerts are representing business user requirements, it is also important that the log management system work hand-in-hand with your email security and policy management solutions. Next Generation Log Management The Sendmail Auditor product overcomes event-data management obstacles and limitations of RDMBS-reliant log management systems. Sendmail Auditor provides the most scalable means to centrally aggregate, efficiently analyze, dynamically monitor and cost-effectively store high-volumes of email and other computer event log data while persevering chain-ofcustody and streamlining forensic investigations. Sendmail Auditor is built upon a modular architecture that takes full advantage of parallel processing, and a clustered repository assuring consistent event collection, analysis and availability. This modular approach allow for appliance-like deployment, distributed configurations and high performance. Sendmail Auditor captures a broad range of event log sources from often dispersed email systems in addition to web proxies, network devices, security applications, host operating systems and applications. Event log data is collected supporting flexible batch and streaming protocols for real-time correlation and complete, long-term historic data analysis. The core of the Sendmail Auditor system is the Scalable Log Server. It provides a scalable, high-speed analytic repository that parses, compresses and executes built-in and user-supplied queries against stored event log data. Sendmail Auditor achieves a 10:1 raw log compression rate, while maintaining full access to all the data for ad-hoc and scheduled analysis. Overall alert monitoring, reporting, investigation and administration are provided by the Analyzer through an intuitive web based interface. The solution is complemented by analytics packages of pre-defined rules and reports, mapped to common security monitoring guidelines and compliance standards. Cost Effective Log Management Because Sendmail Auditor has been specifically designed to solve the data management problems associated with aggregating and analyzing massive volumes of email logs from a variety of sources, it can be accomplished in a pragmatic and cost-effective manner. Sendmail Auditor enables unparalleled precision and long-term search and trending, while significantly saving on storage capacity requirements. The patent-pending data repository supports search against highly compressed email and event logs. Furthermore, clustering technologies provide incremental scalability on load and query throughput, as well as data redundancy and capacity. This scalability is field-proven in some of the largest organizations, accelerating timeto-value through improved productivity that lowers total cost of ownership.
Sendmail Auditor With Sendmail Auditor Business Can: Manage volumes of event data to reduce threat, violation and privacy risks Streamline operational reporting and automate audit processes Accelerate compliance efforts and address data retention guidelines Reduce log management storage, archive, administration and growth costs Readily expand capacity, performance and availability Ensure emails logs are complete, accurate and verifiable CONCLUSION Regulatory compliance, regardless of standard or mandate imposed, requires that organizations monitor all accesses to sensitive data. Typically, sensitive data resides on core applications such as email and is monitored primarily through the analysis of the logs. This data is highly complex and adherence to compliance mandates requires a robust log analysis solution that can accommodate the speed and complexity of these sources for immediate alerting as well as long-term reporting. This represents an enormous data management challenge which Sendmail Auditor is uniquely designed to address. Sendmail Auditor solves the data management problems associated with this class of data, enabling corporations to cost-effectively meet evolving technological and regulatory challenges over time.
MESSAGING BEST PRACTICES EXPERTISE Sendmail s messaging experts can help you with your logging, auditing and email security strategy, best practices, solutions and implementation and support. Large enterprises in 33 countries, including most of the Fortune 1000, trust Sendmail to shield users from unwanted messages, defend the messaging infrastructure, stop data and privacy leaks, and effectively manage messaging to maintain brand and shareholder value, and support regulatory compliance. SENDMAIL EMAIL SECURITY PRODUCTS AND SERVICES To find out more about why businesses are turning to Sendmail to be their trusted messaging advisor, solution provider and implementation support partner, please call: Tel: +1-87- SENDMAIL (877-363-6245) or +1-510-594-5400 (outside U.S.). About Sendmail With 25 years leadership delivering innovative messaging technology, Sendmail ensures the protection and trust of Internet communications. Driven by the industry s most powerful policy engine, Sendmail technology provides protection where 80% of security and compliance violations occur - inbound and outbound messaging. The information provided in this paper is for guidance purposes only and is published as legal analysis, not advice. While every effort is made to provide quality legal and technical information, there are no claims, promises, or guarantees in respect of any specific legal or technical problem. As both legal and technical information must be tailored to specific circumstances, and laws are constantly changing, we recommend you consult a lawyer if you want assurance that the legal information, and your interpretation of it, is appropriate to your particular situation. Sendmail s messaging security experts are also available to review your technical architecture and make recommendations. Included throughout this paper is information taken from other published material written by Sendmail s Logging and Auditing partner, SenSage, Inc.
Sendmail, Inc. 6425 Christie Avenue, Emeryville, CA 94608 Tel: +1 888 594 3150 Fax: +1 510 594 5429 www.sendmail.com 2007 Sendmail, Inc. All rights reserved. Sendmail, the Sendmail logo, Sendmail Directory Services, Sendmail Flow Control, Sendmail Switch, and Sendmail Mailstream Manager are trademarks of Sendmail, Inc. Other trademarks, service marks and trade names belong to their respective companies.