Spam Classification Techniques

Transcription

1 Spam Classification Techniques Executive Overview It costs companies nearly $2,000 per employee a year in lost productivity, double from a year ago. Nucleus Research, USA Today, June 2004 In its infancy, spam was fairly random and easily handled using keyword filters and black lists. Today, there s nothing basic or easy about spam it s tricky, sophisticated and costs corporations millions of dollars every year. Spammers are developing more sophisticated and more dangerous attacks programming their messages and creating threats designed to defeat even the most complex anti-spam and anti-virus filters. By creating blended threats those that combine mass-mailing mechanisms with worms and viruses malicious code can more easily and quickly enter office networks and exploit system vulnerabilities. Studies indicate that in 2005, percent of will be spam, viruses and other unwanted content. 1 Along with the mere frustration of cleaning up after junk attacks is the cost. According to a June 2004 USA Today article, Nucleus Research reported that spam cost companies nearly $2,000 per employee a year in lost productivity a figure that has doubled since In direct response to this increase in spam volume and sophistication, the methods to control threats are also evolving. Some are built into Message Transfer Agents (MTAs) that process mail, some are available through open source software solutions, and some are proprietary solutions created by companies specializing in Internet or security. Unfortunately, spam is a problem that is not going away. And fighting spam requires constant threat monitoring and protection. As new anti-spam techniques are developed to control the latest spam threats, spammers make every effort to stay one step ahead developing newer and even more malicious spam tactics. It is critical that organizations understand the latest spammer tactics and spamfighting techniques in order to combat spammers relentless efforts. Only by remaining aware of and implementing the latest spam protection methods can an organization mitigate the costs and risk associated with spam. 1 Ferris Research 1

2 Today, the most effective methods of detecting, classifying and controlling spam are: Multi-layered filtering is the most effective way of fighting spam. 1. Domain level black and white lists 2. Distributed black lists 3. Heuristics engines 4. Statistical classification engines 5. Honey pots 6. Peer-to-peer networks 7. Challenge-response 8. Reputation analysis 9. Sender authentication This white paper considers these methods of classifying and controlling spam, and suggests that multi-layered filtering is the most effective way of fighting spam using a solution that aggregates a number of these methods into a coordinated spam-fighting framework. Spam Classification Techniques The most effective methods for identifying, classifying and controlling spam today are outlined below. 1. DOMAIN-LEVEL BLACK AND WHITE LISTS Domain-level black and white lists are the original and most basic forms of blocking spam. Employing this method, a domain administrator simply puts an offending spammer s address on a blacklist so that all future from that address is denied. Alternatively, to ensure that from a particular sender is always accepted, administrators would add a sender s address to a white list. While black and white lists can be very useful in blocking or allowing specific addresses, this approach has its limitations. First, it is cumbersome and time-consuming requiring constant list maintenance in order to be effective. Second, spammers use hundreds or even thousands of different addresses to send out their mass mailings, so blocking only a few of these addresses is unlikely to have any significant impact on the flow of spam. Third, spammers often spoof their address, which makes it appear as though their junk is coming from a legitimate sender. In short, while developing black and white lists is a good first step in the fight against spam, this method alone will typically stop less than 10 percent of spam. 2. DISTRIBUTED BLACK LISTS 2

3 Distributed black lists take domain-level black lists to a higher level. Distributed black lists, such as those supported by the Mail Abuse Prevention System (MAPS), operate at the network level. Known spammer addresses and domains are catalogued, and that information is made available on the Internet, either free or by paid subscription. Once deployed into an anti-spam solution, these black lists will automatically block any sent from one of the known spammer addresses on the list. Many organizations have found these distributed black lists to be very useful. Unfortunately, there are instances where legitimate sender addresses have been erroneously added to the list and require significant effort to be removed. For this reason, organizations that are concerned with false positives, legitimate misidentified as spam tend to avoid using distributed black lists. 3. HEURISTICS ENGINES Heuristics are a series of rules used to score the spam probability an human-engineered rules by which a program analyses an message for spam-like characteristics. These rules may look for multiple uses of phrases like Get Rich!!! or Free Viagra! incorporating hundreds and sometimes thousands of rules in order to catch spam. Operating on a scoring-type system, point values are associated whenever a rule detects a spam-like characteristic. Therefore, a message might get a certain number of points for containing a word like Viagra, more points if it contains a click here link, and even more points if the message includes the phrase click here to unsubscribe. Depending on the parameters established, reaching a certain score would classify the message as spam and cause it to end up in quarantine. While heuristics engines have been quite effective over the years, an increase in more sophisticated spamming tactics has impacted the effectiveness of this method. Today, these rules are often available through open-source projects, or as software that can be purchased by spammers just as readily as by regular consumers. Spammers then reverse-engineer these rules to groom their messages before sending them out making sure that they can pass through the filters largely undetected. Another drawback to using heuristics is the labor required to maintain the rules. Even if a company or organization develops its own heuristics engine, the rules must be continuously updated as new spam attacks occur. 4. STATISTICAL CLASSIFICATION ENGINES Statistical classification has been one of the most effective spam fighting methods. Currently, the most common form of statistical classification is Bayesian filtering, which like heuristics, also 3

4 analyzes the content of an message. Unlike rules-based heuristics engines, however, statistical classification engines assess the probability that a given is spam based on how often certain elements, or tokens, within that have appeared previously in other spam s. To make this determination, statistical classification engines compare a large body (or corpus ) of spam with a similarly-sized corpus of legitimate . Each corpus is examined for tokens chunks of text or header information. Some tokens ( Get Rich!!! ) will appear almost entirely in spam , and almost never in legitimate . Thus, based on the prior appearance of certain tokens, statistical classification engines determine the probability that a new message containing certain tokens is spam. The overwhelming advantage of statistical classification methods is that they are extremely accurate, and require only occasional maintenance. Remarkably, statistical classification methods, such as Bayesian filtering, can essentially learn new spammer techniques with minimal human involvement. Yet even though statistical classification has been very successful in its ability to filter spam, some weaknesses have been discovered. 5. HONEY POTS The honey pot method of classifying spam involves seeding the Internet with dummy addresses. The sole purpose of these dummy addresses, or honey pots, is to attract spam. The that comes into these honey pots is then analyzed, and all instances of spam are recorded in a database. The company using this honey pot approach then provides its customers with a service that compares all of that customer s incoming with the database of known spam caught by the honey pots, and blocks any matches. While the honey pot method is very effective, it is usually reactive detecting spam attacks only after they occur. Therefore the honey pot approach works best when combined with other anti-spam solutions. 6. PEER-TO-PEER NETWORKS Peer-to-peer networks offer another interesting method of controlling spam. In short, these networks provide individual members of such a network with the capability (usually a plug-in on their client) to report instances of spam back to the network. Each time a spam appears, the individual members can simply click a button that will, in effect, delete the as spam then simultaneously report the spam back to the network so it is recognized and blocked before it reaches others 4

5 on the same network. Thus, this collaborative network can benefit from a collective effort to control spam. The obvious problem with this method is in the subjective nature of spam classification: what one person considers spam, another may not. And even though these peer-to-peer networks have mechanisms in place to protect against both draconian and overly permissive spam controls, problems with this method remain not uncommon with an approach that involves a collaborative network of users with a multiplicity of opinions. 7. CHALLENGE-RESPONSE This method of controlling spam is a modified and hyper-extended form of the white list. Like a traditional white list, only from senders already on a list of allowed senders is permitted to enter the inbox. In order for a new sender to make it onto the white list, the sender of a new must prove that the message is being sent from an actual human, not a message coming from an anonymous or autonomous spammer s computer. When an comes in from an address not on the recipient s white list, a challenge is immediately sent back to the sender with a message stating that the will only be delivered to the recipient if the sender successfully completes a challenge. These challenges are typically very simple answering a question like How many cats are in this picture? or being asked to Retype the distorted word in the image below. The questions, however, are designed so that they would only be correctly answered by a human. If the sender successfully passes the challenge, the is delivered, and the sender s address is added to the recipient s white list. While this authenticated sender method is certainly effective at blocking spam delivered by the millions from an unknown computer, many users have found the extreme restrictions of this white list-only approach to be frustrating and often unprofessional. And, with systems based on this method, the anxiety and uncertainty of missed always exists since there are times when legitimate is not delivered because the challenge is not completed whether it s because the sender is unable to answer the challenge question immediately, or because an executive feels silly viewing a picture of kittens and checking a box before his or her can be delivered. Furthermore, the recent wave of worms (Sobig, MyDoom, etc.) that can infect PCs and use them to distribute spam are often able to exploit these challenge-response mechanisms. Worms on infected PCs will hijack the address books on local hard drives and send spam to any addresses they find; if recipients of that spam have the infected PC on their allow list of trusted senders, the spam will bypass the challenge-response defense. 5

6 8. REPUTATION ANALYSIS Reputation analysis blocks spam originating from a specific Internet Protocol (IP) address or domain based on the reputation of that IP address. In other words, reputation analysis monitors IP addresses and the amount of spam sent from these addresses. If an IP address sends an enormous amount of spam, then it is usually recognized as a spam source and future attempts of that IP address to send mail will be blocked. The only problem with reputation-based analysis when used alone to stop spam is that IP addresses known to deliver volumes of spam can also send legitimate mail. Thus, reputation analysis must be mitigated with other spam-detection techniques. If reputation analysis is used as the sole method of stopping spam by simply not accepting from offending IP address it increases the risk that legitimate will be blocked. Additionally, the concept of reputation-based filtering can also be applied to anti-spam black lists, wherein black lists that block known spammers can themselves be given a reputation based on how accurate they have proven themselves. 9. SENDER AUTHENTICATION Sender authentication is a new method for blocking spam that attempts to stop spammers from spoofing legitimate domains. In order to avoid detection, spammers will modify their messages so it appears their junk is coming from trusted domains, when in fact the true source (IP address) is very different. In order to protect against this type of spoofing, sender authentication will allow all legitimate domains to provide a public list of the IP addresses associated with their domains. Thus, sender authentication technologies double-check or authenticate every that comes through, making sure that the source IP address is legitimately associated with the domain of the . If the actual IP address on the incoming does not match the IP address associated with the domain, the will be blocked. Sender authentication methodologies, of which SenderID and Sender Policy Framework (SPF) are the most likely to be implemented, are just beginning to be integrated and adopted, and will require an Internet-wide effort to work effectively. Nevertheless, sender authentication is seen as one of the most promising anti-spam technologies proposed, and it has been adopted by a number of spam-fighting companies. Summary of Spam Classification Techniques The above methods of identifying, classifying and controlling spam are not the only ones available to fight spam, but they are currently 6

7 the most common. Indeed, there are other methods, ranging from sophisticated header analysis to copyright-imbedded text that only legitimate senders can use without risking a lawsuit. In the current war on spam, however, the methods described in this white paper have proven to be the most effective. So which of these is the single best method? The only accurate answer would be none. Each one has shortcomings, and none of them are 100 percent effective at blocking spam while maintaining an acceptable false positive rate. Even the Bayesian statistical classification method arguably the current best-of-breed approach to spam control is not perfect. The best method for fighting spam is one that incorporates a number of these nine individual methods. However, incorporating a variety of these methods within one organization might involve a prohibitively complex and expensive effort. Some of these solutions are usually provided as client- or server-side software. Other solutions come in the form of appliances installed alongside a firewall. Still others are provided as services. To combine a number of these into one spam-fighting solution would, therefore, require an unwieldy, patchwork system of approaches, and managing them could become a full-time job for an organization s IT administrator. For this reason, most organizations choose one type of filtering method and work with its limitations. The best method for fighting spam is one that incorporates a number of the nine individual methods described in this white paper. There is, however, a means of combining a number of these solutions into one coordinated spam-fighting solution controlled by a single interface. Only as a managed service can multiple spam control solutions be incorporated into a unified spam control system using a single interface. MX Logic has created exactly this sort of solution in its Stacked Classification Framework. MX Logic s Stacked Classification Framework No longer considered merely a nuisance, unwanted, unsolicited is putting extreme pressure on employee productivity and network capacity across organizations worldwide. To help defend business networks with the most accurate spam filtering and protection in the market, MX Logic developed its Stacked Classification Framework. This patent-pending technology combines the most effective spam-fighting techniques in the industry to successfully block 98 percent of spam with an industry-low rate of misidentified as spam also known as false positives. 7

8 The Stacked Classification Framework uses the combined filtering power of techniques including Bayesian Statistical Analysis, Reputation-based Filtering (for both IP addresses and Realtime Blackhole Lists), Industry-wide and Proprietary Heuristics Rules- Based Analysis, Reputation Analysis, URL Filtering and Reputationbased real-time blackhole lists (RBL) filtering. (Outside of the Stacked Classification Framework, MX Logic also uses Distributed Black Lists and Global Black Lists, and Customer-Defined Deny and Allow Lists to further identify spam.) And, MX Logic will soon be integrating Sender Authentication. Each of these filters dynamically calculates the spam probability of every message. With this multi-layered approach, the different spam filters can separately assess and vote on the probability that a specific is spam. Because each filtering method has unique strengths designed to identify specific threats, the combination creates one of the most accurate and comprehensive filtering processes in the industry. The MX Logic Defense Service use a multi-layered spamfighting process to identify unsolicited junk . It provides control over spam before it can reach the enterprise network. Enterprises can set specific filtering parameters based on their established corporate policies, which include deleting, tagging, or allowing specific messages and constructing customizable black and white lists. While MX Logic boasts one of the industry s lowest false positive rates, organizations can further reduce misidentified by using the sophisticated quarantine features to define how quarantined messages should be handled in the future. In addition to the Stacked Classification Framework, the effectiveness of MX Logic s solutions is made possible by the integration of unique and advanced technology and processes including powerful perimeter-based protection, in-line message filtering, scalable and robust architecture, virus and worm engines, and around-the-clock threat management. With MX Logic s comprehensive anti-spam technologies, which provide the most advanced spam-fighting techniques available, enterprise networks remain free from offensive, unwanted, and unsolicited . About MX Logic MX Logic, Inc. provides innovative defense solutions that ensure protection and security for enterprises, service providers, government organizations, and resellers and their customers. Deployed as a managed service or on-premise software, 8

9 the company's feature-rich solution suite is the industry's most comprehensive, flexible and easy to use. MX Logic s cost-effective service provides around-the-clock protection, automatically intercepting, analyzing and blocking malicious and unsolicited messages at the network perimeter before they can enter or leave an internal network. Unlike other protection solutions, MX Logic s services act as a proxy, filtering messages in-line as they are delivered to the customer reducing the risk of message loss common to the store-and-forward method used by other providers. Using a patent-pending Stacked Classification Framework, which leverages the strengths of leading spam-fighting filters, MX Logic can accurately stop 98 percent of spam and viruses at the network perimeter. Fortifying the filtering process with two industry-leading anti-virus engines makes our Defense Service one of the most comprehensive solutions on the market today. Through the company's managed service offering, MX Logic processes millions of messages per day for over 2,500 organizations, including EnCana, Hyundai Motor America, The Sports Authority, YMCA, and ServiceMaster. In addition, MX Logic is the only defense company to offer both a managed service and a turnkey, carrier-grade software solution for service providers. For more information, visit. 9