Why Content Filters Can t Eradicate spam

Transcription

1 WHITEPAPER Why Content Filters Can t Eradicate spam About Mimecast Mimecast () delivers cloud-based management for Microsoft Exchange, including archiving, continuity and security. By unifying disparate and fragmented environments into one holistic solution that is always available from the cloud, Mimecast minimizes risk and reduces cost and complexity, while providing total end-toend control of . Founded in the United Kingdom in 2003, Mimecast serves over 5,000 customers worldwide and has offices in Europe, North America, Africa and the Channel Islands. For more information, please visit or info@mimecast.com.

2 Contents 03 Defining the Problem 03 Current Spam Management Systems 04 Implications of Content Filtering for Spam Capture 04 Quarantining 04 False Positives 05 Connection Filtering 05 The Mimecast Approach to Spam ARMed SMTP 2

3 Defining the Problem Spam means different things to different people. Some people define any unwanted or unsolicited as spam (even if they asked for it but don t want it anymore) while others consider all bulk s (both solicited and unsolicited newsletters and marketing announcements) to be spam. Many people label any from people they don t know as spam. This range of perceptions, coupled with spammers ability to vary message content to appear more or less like spam, means it is difficult for a software product to completely eliminate it. As a result, the fight against spam has become an arms race with spammers and Secure Gateway vendors trying to outfox one another. Legal authorities have engaged in different activities to address the issue, from shutting down sympathetic hosting providers, to working behind the scenes to kill botnets. While not wholly effective, the process has at least forced unscrupulous senders to operate from the shadows to avoid the risk of legal prosecution and more interestingly, it has provided mechanisms for simpler identification of friend from foe. Recent statistics suggest that spam outweighs legitimate by more than four to one. Practically all businesses now have spam management filters in place, but most have not yet been able to find a solution to the problem, and often the implementation of a spam filter has merely moved the spam problem and created a number of other message delivery issues. Current Spam Management Systems The vanguard of current spam management systems has been the content filter. These spam filters use various policy driven techniques applied to the words and body content of in order to determine whether the should be delivered to the recipient or not. The most common content filtering techniques employ Heuristic analysis and Bayesian analysis. We can loosely describe the technical process involved in applying these techniques as analyzing of the words or makeup of s in order to detect a pattern, then comparing that pattern to a database of known bad or malicious content. Examining words or phrases commonly used in spam is a simple approach to the problem, but is not completely effective because it assumes that those words and phrases will not appear in legitimate s. More sophisticated techniques apply weighting scores to known bad words or phrases and if the total score for the exceeds a certain limit, then it considers the to be spam. Some content filtering systems try to learn about what types of s the recipient prefers as distinct from those that are rejected most others require a regular update of known bad phrases and settings. Some security companies have even gone as far as to suggest that their systems are close to artificial intelligence. But, the arms race still continues as the spammers learn how to circumvent the latest techniques designed to defeat them. 3

4 Implications of Content Filtering for Spam Capture The problem with content filtering is not that it cannot block spam, but rather that it is the most expensive method, in terms of technical processing, as well as the cost of deploying and maintaining that solution. Content filtering deals with an infinite number of variables, which makes it error prone. However, the key failure in the process is the flawed assumption that a software program can determine what s an individual would like to receive. Systems using content as the only consideration are easily broken when people subscribe to useful and legitimate newsletters which have much in common with spam s. Many legitimate s will also be marked as spam by a content filter so called false positives - if they include common spam words and phrases. This issue is a particular problem in certain industries such as healthcare and financial services where their keywords are frequently the subject matter of real spam. Quarantining To try and combat the problem of false positives that is symptomatic of content filters, vendors added another break in the SMTP chain, called quarantining. Almost all security systems available on the market today use a quarantine folder to store all s marked as spam. administrators have traditionally been required to manually sift through these quarantines on a daily or hourly basis looking for false positives. Often, the first indication that an has been mistakenly identified as spam is when the sender calls to ask why there has been no response to an . In a business context, that can mean lost sales and clients. Some solutions require the end user to review their own quarantine folders in order to locate incorrectly classified s. In this scenario, they may as well receive the spam directly because the time required to scan the quarantine is the same amount to view an overburdened inbox. Quarantines create new problems without solving the existing ones. They merely move delivery problems around, placing an additional burden on the administration team or the end user. False Positives The more aggressive a spam filter becomes, the more likely it is to reject legitimate s along with spam s. This condition is called a false positive. vendors have traditionally focused on the idea that NO spam should get through to the end user. However, this approach does not take into account the fact that a single , incorrectly blocked, is many times more costly to deal with than one junk received. It is therefore incorrect to assume that 1% or even 0.1% of s incorrectly classified as spam is acceptable. administrators have begun to realize that the cost of false positives greatly outweighs the cost of dealing with the spam itself. False positives represent a breakdown in communication, lost opportunities and productivity, and mistrust of as a communications medium. 4

5 Connection Filtering More advanced Secure Gateway vendors, particularly those delivering their service from the cloud are able to offer an additional level of protection before the is processed by the content filtering engines. Connection filtering seeks to examine the source, and in some cases the destination, of the to determine whether or not the sender has a reputation within that system already. Reputations can be classified into a number of groups. Globally Known Bad would imply the sender already has a bad reputation maintained on the wider Internet, so would be listed by their IP address in an RBL or Realtime Block List. Locally Known Bad would imply the anti-spam solution maintains its own database of known bad senders; some administrators achieve this manually with organization-wide block lists. Good reputations would be classified as Locally Known Good, where the Anti-spam solution is automatically maintaining a database of known good communication pairs, i.e. the addresses and IP addresses your users regularly send to. Anything received from those external contacts can be assumed to be Known Good. Building a reputation with a Secure Gateway is also possible, provided that the gateway supports RFC compliance checking techniques like Gray Listing. Passing such a test implies that whatever is sending an has queued and retried at RFC compliant intervals, so is likely to be a legitimate SMTP Server. The Mimecast Approach to Spam ARMed SMTP Mimecast precisely and correctly blends advanced reputation and protocol connection techniques into a powerful and effective anti-spam system. This sophisticated capability is the result of the Mimecast platform s market-leading Mail Transfer Agent (MTA) architecture and the unique in-protocol and connection level anti-spam tests that we apply. While other vendors have supplemented their legacy content filtering approaches with some standard connection filtering features to shore up their solutions, Mimecast ARMed SMTP centers on this progressive methodology, reducing the reliance on content examination techniques for spam detection. Mimecast can intercept the majority of spam without examining the body content of an because our experience and our technology lets us identify patterns of SMTP delivery behavior that are typically exhibited by spammers. We identify these patterns by the way spammers deliver their s and not according to the content of the s themselves. This approach offers profound benefits. Mimecast leaves spam undelivered with the spammers so there is no local bandwidth loading; it significantly reduces the overhead of managing quarantine folders and the integrity of the SMTP protocol is maintained so that no legitimate messages go missing. ARMed SMTP is a way of dealing with s and spam, developed for the way is used today, rather than how it was originally expected to be used twenty years ago. Our approach provides a more successful, robust, cost effective and intelligent solution to the problem of spam and spam management Mimecast. ALL RIGHTS RESERVED. WHI-WP