MailMarshal SMTP Anti-Spam Configuration

Transcription

1 MailMarshal SMTP Anti-Spam Configuration November, 2006 Contents Best Practices 2 Optimum Base Configuration 2 SpamCensor 3 URLCensor 6 CountryCensor 16 HELO Rules 20 TextCensor Scripts 21 Attack Prevention 24 Spam Management Tips 25 Whitelisting Practices 28 Submitting Missed Spam to Marshal 34 Common Pitfalls 35 Conclusion 36 MailMarshal SMTP provides an excellent spam detection rate out of the box. Basic configuration settings are covered in the white paper MailMarshal SMTP Anti-Spam Basics. You should read that paper and review your MailMarshal SMTP configuration before continuing to this paper. This paper provides detailed information about how to adjust the settings of MailMarshal SMTP 2006, the SpamCensor, the CountryCensor, and the other available Category Scripts. This paper requires an intermediate level technical understanding of concepts and MailMarshal configuration. To fully understand and apply the ideas in this paper, you should be familiar with the MailMarshal Configurator, the registry editor, and text configuration files. Author: Chris Harris 1

2 MailMarshal SMTP 2006 continues the tradition of providing a comprehensive tool to control spam based on an extensive array of functionality. The two key concepts are detection and management. MailMarshal SMTP uses technologies that enable high spam detection rates and few false positives, with easy administration and a variety of precise customization options. It does this within the context of an integrated content management package. MailMarshal SMTP is more than an anti-spam system it provides organizations with the means to control all content, including spam, viruses, text, and attachments, within a rules-based framework. Best Practices Whether a system has been installed cleanly with all of the default rules in place, or upgraded from a number of major versions back, it is very important to ensure that MailMarshal SMTP is taking advantage of all of the features available. Spam updates are retrieved automatically through the web, but Marshal will not make modifications or additions to existing rules. In order to ensure that you are using the latest technology released, and to make use of the files downloaded through the automatic updates, a base configuration should normally have a number of Category Script driven features enabled. These include: SpamCensor URLCensor URLCensorIP DNS blacklists CountryCensor A number of other functions available within MailMarshal SMTP can also be used for anti-spam purposes. These include: HELO rules, which are used to examine the behavior of the connecting SMTP system Attack Prevention capabilities, which allow you to enforce network-friendly behavior on the part of connecting systems TextCensor scripts, which provide you with a simple way of updating spam detection capabilities on the fly before automatic spam updates are released. The following sections will discuss the use of the various Category Scripts and other techniques available within MailMarshal SMTP, and provide instructions for their implementation. Optimum Base Configuration To achieve the highest catch rate and optimum performance, anti-spam rules should be run in the following order: SpamCensor URLCensor DNS blacklists URLCensorIP CountryCensor (optional: multi-national companies should have a comprehensive exclusion list when using CountryCensor) When SpamCensor, URLCensor, URLCensorIP, and DNS blacklists are used in this order, most environments will see a spam catch-rate of over 99.5%. This document will cover more than just the aforementioned rules, but at a bare minimum these should be enabled. Adding CountryCensor to the equation can contribute to an even higher spam detection rate. Note: For basic information about these essential rules, see the Anti-Spam Basics white paper. In order to maximize the effectiveness of these components, the primary MX record for a domain should point directly to the server on which MailMarshal SMTP resides, rather than directing through a forwarder, SMTP proxy, or 2

3 relay of any sort. Many powerful checks used by SpamCensor and the other rules depend upon a remote host s initial communication with MailMarshal SMTP. Ensuring a direct MX connection is essential to ensure the effectiveness of any DNS Blacklist checks that are performed within Receiver rules, because the blacklist checks query the IP address of the connecting host. SpamCensor The SpamCensor is an advanced heuristic filter that utilizes a combination of techniques to identify spam. Much more than a simple key word filter, it utilizes the following: Detailed header analysis. This technique closely examines message headers for any indication that it may be spam. The SpamCensor looks for hundreds of typical spam indicators. These include irregularities such as, missing To or From header fields, invalid dates, and spaces in unusual places; they also include typical traits or spam genes usually left by bulk mailers or spamware the tools used to create and send spam. Advanced analysis of message content. The SpamCensor filter performs advanced searches of message content. It searches for thousands of spam patterns, including common phrases that promise get rich quick schemes and a better sex life, words with gaps between the letters, and sophisticated HTML patterns known to be associated with spam. It has rules that target different areas of each message, including plain text, raw html, and URL links. It can scan anything from the text between HTML tags, to the contents of the HTML tags themselves. Message composition. SpamCensor checks the message size and composition. Spam is not typically large, and often has only an HTML part. This information is used alongside numerous other indicators. As the SpamCensor runs, the results from each of the thousands of tests contribute to an overall spam picture. Each item contributes to a numeric score. Once the score exceeds a threshold, MailMarshal SMTP will treat the message as spam and take a predefined action. This weighted score approach results in high spam detection rates with few false positives. Using the SpamCensor The following files are referenced in the discussion of SpamCensor: SpamCensor.xml SpamChecker.dll SpamEvals.dll spamfilter.xml UserDefined.xml These files are found under the Config directory within the MailMarshal SMTP installation path. In a fresh installation of MailMarshal SMTP 2006 the installation path is C:\Program Files\Marshal\MailMarshal\Config\ Before using the SpamCensor functionality, you should ensure that MailMarshal SMTP is using the latest revisions of these files by performing a Spam Update (in the Configurator, see Tools > Server and Array Properties > Spam Updates). If the updater is unable to check for updates, please contact Marshal Technical Support. Category Scripts Category Scripts are XML configuration files which contain different types of rules for checking . The SpamCensor is a special type of Category Script. As of MailMarshal SMTP 5.5 and later, a new Rule Condition has been introduced, titled Categories. This condition uses Category Scripts to check messages. The default MailMarshal SMTP rules now make use of SpamCensor. If your installation has been upgraded from an earlier version than 5.5, SpamCensor may not be in use. 3

4 To enable the SpamCensor you must create rules that use the SpamCensor Category Script. If your MailMarshal was first installed at version 5.5 or later, a SpamCensor rule should already exist. Basic Configuration SpamCensor is designed to be simple to set up, and once enabled in a rule it will immediately begin catching spam. This section discusses basic configuration of the SpamCensor. Although there is a range of more advanced adjustments that can be made, in most cases the basic configuration is all you need. Note: Before creating a SpamCensor rule, check that one does not already exist. To enable the SpamCensor: 1. Start the Rule Wizard by right-clicking an existing Policy Group, and selecting New Rule. 2. Choose Standard Rules and select Next until you arrive at the Rule Conditions window. Select the checkbox Where message is categorized as. 3. Create a rule that uses the SpamCensor.xml file. You will see a window as below. You will notice in the window that there are other XML Category Script files to choose from, some of which may not exist within your installation. These may include other standard scripts, and custom scripts. Warning: You should select only a single Category Script in each rule. Multiple Category Scripts should NOT be checked in this window. Selecting multiple scripts significantly reduces the effectiveness of a rule. If multiple scripts are selected the rule is evaluated using an AND condition. ALL selected scripts must return positive for the condition to be true. For example in this instance, if both SpamCensor AND SpamCop were checked, the rule would only evaluate true for messages that trigger BOTH on SpamCensor, AND on IP addresses blacklisted on SpamCop. 4

5 4. You should finish with a rule that looks similar to this: Standard Rule: Block Spam - SpamCensor When a message arrives Where message is incoming Where message is categorized as 'Spam' Move the message to 'Spam' You can use all the usual rule elements to refine your rules. For instance, you can combine a whitelist and a size rule to improve accuracy. The whitelist would typically contain lists of newsletter sources, or other trusted or key sources of bulk . MailMarshal SMTP can even be configured to automatically generate a whitelist of friendly senders by harvesting recipient addresses on outbound s. Size conditions could be added as well, which would eliminate scanning of larger s that are unlikely to be spam. Note: Any refinements, particularly size conditions, should be reviewed regularly to ensure they are not reducing the effectiveness of SpamCensor. A note on False Positives Whitelists are an important tool to reduce false positives. The SpamCensor is a heuristic filter that seeks to identify unsolicited bulk . Wanted bulk can be difficult to distinguish, since users may disagree about which messages are wanted. A comprehensive list of friendly addresses not only ensures the successful receipt of wanted , but also has the additional benefit of allowing the filters within MailMarshal SMTP to be stricter than is feasible within a default setup. Note: Over-use of whitelists, especially the use of wildcards, can contribute to false negatives. In particular, whitelisting your own domain allows significant amounts of spam to pass through. Some automated tactics will be detailed later in the Whitelisting Practices section. Administrators should also encourage and train their end-users to make use of the web-based Spam Quarantine Management system. This system allows each user to create personal white and black lists. Since this document is primarily concerned with fine-tuning anti-spam filters, setup and configuration of the Spam Quarantine Management system is not covered here. Additional information regarding the Spam Quarantine Management system can be found in the MailMarshal SMTP User Guide, or by contacting Marshal Technical Support. Reviewing the SpamCensor Result You can examine the log file in the MailMarshal Console for the reason why a particular message is blocked. In order to find the message in question, you can either attempt to locate the message in Mail History, or perform a search based upon relevant criteria. Once the message is found, click the tab labeled Log, and you will see an excerpt like the one below: 5

6 SpamCensor Logging Levels By default, MailMarshal SMTP does not retain a record of the SpamCensor score for messages that are not blocked. When testing the SpamCensor it is sometimes useful to know what rules triggered when a message did not reach the trigger level. The following Registry setting causes the SpamCensor to always log its output. Open regedit on the Array Manager server, and navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Marshal\MailMarshal\Default\Engine Add the following registry entry: Name: LogSpamAlways. Type: DWORD Value: 1 Set the value to 1 (true) to enable this extra logging. Setting the value to 0 (false) will disable the extra logging. This setting does not significantly affect server load, but it does add volume to the text logs. URLCensor URLCensor queries external DNS blacklists which provides records of domains that appear to be frequently advertised within spam messages. (These lists differ from traditional DNS blacklists, which list individual IP addresses.) The original purpose of this functionality was to provide a method of blocking messages that contained very few triggers other than a link to a notoriously spam-advertised domain. Over time it has proven to be an excellent complement to SpamCensor. How does URLCensor work? URLCensor parses the body of an , and extracts all the domain portion of any URLs that are found. It then performs a DNS A record lookup for each domain using the DNS blacklist. There are currently two permutations of URLCensor available for use within MailMarshal SMTP: URLCensor URLCensorIP URLCensor - Checks domains found within a message body against a third-party DNS blacklist. By default URLCensor uses the blacklist maintained by SURBL.org, or more specifically, multi.surbl.org. It can easily be configured to use other blacklists as well (covered further on in this document). multi.surbl.org is a 6

7 combined zone utilizing domains provided by SpamCop, abuse-butler, SpamAssassin, and others. If for example URLCensor were to query marshal.com against SURBL.org, it would query the A record for marshal.com.multi.surbl.org. By default, if the DNS query returns any record at all, URLCensor will consider the domain to be blacklisted. If no record is returned from the blacklist s DNS server, the domain is not considered to be blacklisted. Once the lookup is performed, the result, whether positive or negative, will be cached by URLCensor for a certain (adjustable) time to preserve performance and avoid the need for repeated DNS lookups. URLCensorIP - Performs in a similar way to URLCensor, but is designed to query against a blacklist that is formatted by the IP address of the A record for the domain, rather than by the domain name. URLCensorIP resolves the domain to an IP address using a traditional DNS query, and then submits the DNS blacklist query. By default, URLCensorIP uses the combined Spam and Exploits blacklists maintained by Spamhaus. Again if any result is returned, the domain is considered to be blacklisted. If no record is returned, the domain is not considered to be blacklisted. URLCensorIP caches the results of these queries for a specific interval in case they need to be used later. Querying the IP address instead of the domain name is useful because spammers register large numbers of new domains, and thus domain blacklists are difficult to keep up-to-date. However, because the spam-related domains typically use a much smaller number of IP addresses, it is easier for the IP based blacklists to maintain a good hit rate. Both URLCensor and URLCensor IP can be configured to query other third party blacklists, so long as they are in one of the two supported formats. Both also have a configurable cache duration. For more information, see the White Paper MailMarshal SMTP Anti-Spam Advanced Configuration. Using the URLCensor The following files are referenced in the discussion of URLCensor: SpamSurbl.dll URLCensor.xml URLCensorIP.xml These files are found under the Config directory within the MailMarshal SMTP installation path. In a fresh installation of MailMarshal SMTP 2006 the default installation path is C:\Program Files\Marshal\MailMarshal\Config\ Before using the URLCensor functionality, you should ensure that MailMarshal SMTP is using the latest revisions of these files by performing a Spam Update. If the updater is unable to check for updates, please contact Marshal Technical Support. The URLCensor is intended to be simple to implement. This section discusses basic configuration of the URLCensor. In most cases the basic configuration is all you need. However, for those who like experimenting, a range of advanced adjustments can be made (see the White Paper MailMarshal SMTP Anti-Spam Advanced Configuration ). To enable the URLCensor, create rules that use the URLCensor and URLCensorIP Category Scripts. In new installations of MailMarshal 2006 these rules are present by default. These rules make use of the same Categories Rule Condition that is used for SpamCensor and all other Category Scripts. To enable URLCensor: 1. Start the Rule Wizard by right-clicking an existing Policy Group, and selecting New Rule 2. Select Next until you arrive at the Rule Conditions window. 3. Select the checkbox Where message is categorized as. 4. Create a rule that uses the URLCensor.xml file. You will see a window as below: 7

8 5. Select the checkbox for URLCensor.xml 6. Complete the wizard by naming the rule, and deciding upon an appropriate action. Many actions can be taken based upon company policy and what the Administrator deems appropriate. For instance you can move the message to a folder, or simply flag the message for handling by the end-user s mail client. You should finish with a rule that looks similar to this: Standard Rule: Block Spam URLCensor (by Domain) When a message arrives Where message is incoming Where message is categorized as 'URLCensor Blacklisted' Move the message to 'Spam' As with SpamCensor, you can use all the usual rule elements to refine your rules. You can add a whitelist, a size rule, a TextCensor excluding certain domains, and so on. To enable URLCensorIP: 1. Start the Rule Wizard by right-clicking an existing Policy Group, and selecting New Rule 8

9 2. Select Next until you arrive at the Rule Conditions window. 3. Select the checkbox Where message is categorized as. 4. Create a rule that uses the URLCensorIP.xml file. You will see a window as below: You should finish with a rule that looks like this: Standard Rule: Block Spam URLCensor (by IP Address) When a message arrives Where message is incoming Where message is categorized as 'URLCensor IPBlacklist' Move the message to 'Spam' URLCensor and False Positives Whitelists always play an integral role in preventing false positives. However, because of the nature of the URLCensor checks, false positives are very rare. The team at Spamhaus and SURBL do an excellent job of maintaining these lists and keeping them up to date to ensure that legitimate domains do not get listed, and that spam-advertised domains are listed in as timely a manner as possible. 9

10 Reviewing the URLCensor Result You can examine the log file in the MailMarshal Console to determine the reason why a particular message is blocked by URLCensor. You will see an excerpt like the one below. The log file illustrates how the URLCensor works. From this log, you can see that the domain (j4fimage.com) is blacklisted on multi.surbl.org. This particular domain exists in the DNS blacklist maintained by SURBL.org, indicating that, SURBL.org had received indications that this domain was a commonly spam-advertised domain. DNS Blacklists In addition to examining domain names (URLs) found within a message body, MailMarshal SMTP can examine the list of servers through which a message has traveled to see if any of them are known spam sources. The IP addresses found within Received lines of a message header indicate the servers through which a message has traveled. As the services use DNS as the method of querying their servers, they are also often referred to as DNS blacklists. There are quite a number of blacklists available on the Internet. The lists vary in quality, availability, and aggressiveness of listing policies. These lists are usually maintained by non-profit organizations, although some charge for certain services. One of the best known commercial sites is the Mail Abuse Prevention System Real-time Blackhole List (MAPS RBL) now owned and managed by Trend Micro. Some of the well known services are: MAPS ( SpamCop ( SPEWS ( Spamhaus ( Each of the various blacklists has its own criteria for determining the contents of their respective lists. Before adding a new DNS blacklist, you should read the listing policy, if it is public, and speak to other users to determine the likelihood of false positives. The two most popular DNS blacklists in use today are those provided by SpamCop and Spamhaus. For more information on listing criteria and the lists in general, see and MailMarshal SMTP can perform queries against these blacklists, to aid in the determination of whether or not a message should be considered spam. MailMarshal SMTP 2006 provides rules to query both SpamCop and Spamhaus by default. If your installation has been upgraded from an older version, you may need to create the rules. 10

11 This section will deal solely with SpamCop and Spamhaus. Other lists can be easily integrated. This process is discussed in the White Paper MailMarshal SMTP Anti-Spam Advanced Configuration. How can MailMarshal SMTP use DNS Blacklists? MailMarshal SMTP can perform DNS blacklist lookups of IP addresses in two ways: 1. Receiver Rules 2. Standard Rules (using Category Scripts) Each of these two methods has a specific purpose. For optimal performance and anti-spam recognition, it is ideal to complement a Receiver rule that performs a DNS Blacklist lookup with a Standard rule that does the same. The reasons for this recommendation are explained below. DNS Blacklists in Receiver Rules and Standard Rules There are important differences between the behaviors of Receiver rules and Standard rules in the use of DNS blacklist lookups. Both rule types have their merits and drawbacks, and ideally both should be used. Receiver-based DNS blacklist lookups Receiver rules that utilize DNS blacklist lookups query the IP address of the connecting host. This is one of a number of reasons MailMarshal SMTP should be the gateway of the network (the first server that accepts a message when it enters the network). Receiver-based DNS blacklist lookups are rendered useless if another gateway is placed ahead of MailMarshal SMTP. In that case the connecting IP address will always be the same (the IP of the other gateway). The IP address of the external server that connected to this gateway might be blacklisted, but the MailMarshal Receiver has no information beyond the server that connected directly to it. Even if a message originated from a known spam source, a Receiver rule will never trigger because the connecting server is NOT listed as a known spam source. Another issue can occur if the MTA of your ISP is designated to handle inbound prior to passing it on to MailMarshal SMTP. In the rare event that the MTA of your ISP is listed on a DNS Blacklist, all will be rejected by the Receiver. Because Receiver rules reject a message rather than simply quarantining it, if a legitimate message is inadvertently rejected at the Receiver, it will never be retried but is returned to the sender immediately. This threat is remote but should be considered prior to enabling this or any type of Receiver rule. On the other hand, this same behavior can provide an excellent benefit in terms of bandwidth and performance. A Receiver rule will reject a message subsequent to the remote MTA issuing the RCPT TO command in the initial SMTP handshake. In this scenario, the actual message body is never transmitted. The benefits of this are twofold: 1. Rejecting a message prior to the sending of the message body can reduce the bandwidth consumed by unwanted, unsolicited messages. 2. Preventing the message from entering the system also prevents it from consuming a MailMarshal Engine thread. Typical installations will have 2 Engine threads (with 4-5 in extreme circumstances on more robust hardware). Any message the Engine does not have to deal with improves performance, and frees the Engine to appropriately process legitimate messages entering the system. Standard Rules performing DNS Blacklist lookups Due to the limitations of Receiver rules, in most instances they should be supplemented with Standard rules. Standard rules use Category Scripts to perform DNS RBL lookups against lists such as SpamCop and Spamhaus as these rules offer more flexibility. Standard Rules using SpamCop or Spamhaus cause MailMarshal SMTP to parse through the Received lines within a message header for IP addresses of servers. Each IP address found results in a query to the DNS RBL. This method of DNS RBL lookup implementation has the benefit that it checks for blacklisting of intermediate servers through which a message has traversed. If any of these servers are listed, the IP address will trigger the rule. 11

12 However, since the entire message is received before a Standard rule is applied, the bandwidth to transmit the message has already been used and an Engine thread will be required to process the message. Important Note on DNS Blacklist Lookups URLCensor, URLCensorIP, SpamCop, and Spamhaus all require frequent requests to be sent to DNS. The DNS server used for these lookups, as well as any other functions within MailMarshal SMTP that require DNS, is the DNS server specified within the Delivery settings in the Configurator. If an array of MailMarshal SMTP servers is in use, by default all nodes in the array will use the Delivery settings defined in the Server and Array Properties. You can specify custom Delivery settings for a node in its individual Server Properties. MailMarshal SMTP will NOT use the DNS server specified in the TCP/IP settings of the server s network interfaces at any point in the message handling. It is absolutely imperative that a responsive, dependable, and forward-resolving DNS server is specified within MailMarshal SMTP s Delivery settings. One of the most common causes of poor Engine throughput and Receiver responsiveness is a setup that uses DNS lookups through a slow or unresponsive DNS server. This problem is especially noticeable when DNS blacklists are used. You can check the time used for each processing action by reviewing the text Engine log. If the DNS server seems to be a source of delay, you may wish to set up an internal, local DNS server using a DNS Zone Transfer for the DNS blacklists in question. The procedure for setting up this configuration lies outside the scope of this document, and will vary depending on the DNS server software being used. DNS Blacklist Server Downtime and Timeouts Occasionally DNS Blacklist servers become unavailable. In this scenario, MailMarshal SMTP waits for a period after a failed DNS Blacklist connection and tests connectivity before resuming full use of the server. Messages will be processed without checking against the DNS Blacklist until the server becomes available again. By default MailMarshal SMTP re-tries a server four times before marking it unavailable. SpamCop/Spamhaus use within Receiver rules Before you enable any Receiver rules that use DNS blacklist lookups, you must enable each blacklist within the Host Validation window on the MailMarshal Configurator. To access this window, open the Configurator and select Tools > Server and Array Properties > Host Validation. This window allows you to confirm which lists are currently enabled for use within Receiver rules: 12

13 Note that in the illustration, no DNS blacklists are enabled. To enable one of these DNS blacklists for use within a Receiver rule, highlight the desired DNS Blacklist and click Edit. This will open a window similar to the one pictured below: Enable the blacklist by checking the Enable box. Once a blacklist is enabled, you can use it within a Receiver rule. To create a DNS Blacklist Receiver rule: 1. Start the new rule wizard by right-clicking the desired policy group and selecting New Rule. 2. On the first pane, change the type from Standard Rule to Receiver Rule and then click Next. 3. If a whitelist of safe senders is available, it should be used. To use a whitelist, on the User Matching pane, add the User Matching condition Except where addressed from. Then click the red users hyperlink and select the user group corresponding to the whitelist, as seen below: Once the group is highlighted, click the middle double - arrow (<<) to add the user group, then click OK. Using a whitelist excludes friendly senders on the list from having mail rejected by this rule. NOTE: Due to the aggressive nature of Receiver rules, it is good practice to exclude a list of known legitimate senders from Receiver rules in general. If a Receiver rule is triggered, MailMarshal SMTP will respond with a 500 series response code, which means that the message is rejected permanently. This code will cause the connecting server to generate a Non Delivery Report (NDR) and return it to the original sender. 4. In the rule wizard, click Next. 5. Select the option Where sender s IP address is listed in DNS Blacklist. 6. On the blacklist selection window, all DNS Blacklists currently enabled within Host Validation are listed. Check the box to select the DNS Blacklist of your choosing and then click OK. 13

14 7. Click OK to continue to the Rule Actions pane. 8. Ensure that Refuse message and reply with is selected. You can customize the response code and brief message sent by clicking the blue Refuse message hyperlink. You should finish with a rule that looks like this: Receiver Rule: Deny Spamhaus Blacklisted Senders at Receiver When a message arrives Where message is incoming Except where addressed from Global Whitelist Where sender s IP address is listed in 'Spamhaus SBL-XBL' Refuse message and reply with ' Rule imposed as {Sender} is blacklisted on Spamhaus (see Using SpamCop and Spamhaus within Standard rules Through the use of Category Scripts, MailMarshal SMTP can utilize DNS Blacklists within Standard rules. Both Spamhaus and SpamCop are available and enabled by default in the latest release of MailMarshal SMTP. If these blacklists are not currently in use, setting them up is as quick and simple as utilizing any other Category Script, such as SpamCensor. To enable Spamhaus and SpamCop checks within Standard rules: 1. Start the Rule Wizard by right-clicking an existing Policy Group and selecting New Rule. 14

15 2. Select Next until you arrive at the Rule Conditions window. 3. Select the checkbox Where message is categorized as. 4. Create a rule that uses the appropriate XML file. You will see a window as below. For Spamhaus: Or For SpamCop: You should finish with a rule that looks similar to this: Standard Rule: Block Spam Spamhaus Blacklisted When a message arrives Where message is incoming Where message is categorized as 'Spamhaus Blacklisted' Move the message to 'Spam' 15

16 Reviewing the SpamCop/Spamhaus Standard Rule Results Using the MailMarshal Console, you can examine the message log file to determine why a particular message was blocked by either the Spamhaus or SpamCop standard rules. Note: Messages rejected by Receiver rules will not be shown in the MailMarshal Console. Analysis of receiver rules will require manual review of the MMReceiver logs. For messages blocked by Standard rules, you will see an excerpt in the Console like the one below: In this instance, note that the IP address being queried ( ) was not listed on sbl-xbl.spamhaus.org, but, it was listed on bl.spamcop.net (the DNS query to bl.spamcop.net using this IP address returned a record). The log shows the IP address in reversed order. This is simply due to the setup of most IP-based DNS Blacklists. CountryCensor Included with MailMarshal SMTP 2006 is a powerful, unique utility called CountryCensor. CountryCensor allows mail administrators to identify the countries through which a message has traveled, and handle it accordingly. This capability can be very useful for an environment that receives little legitimate from countries other than its own, or for environments where from specific countries should be handled in a manner different from others. It is important to note that CountryCensor does NOT look at the top-level domain name found in any part of a message but rather examines the IP addresses in the message header to determine the countries through which the message has traveled. 16

17 Prerequisites CountryCensor requires the following files be in place within the Config folder under the MailMarshal SMTP installation path: CountryCensor.xml CountryCensor.dll CountryCensorGroups.xml ip.db cc.db A default installation of MailMarshal SMTP 2006 will already contain these files. If they are not present within your installation, please contact Marshal Technical Support. Basic Configuration CountryCensor currently requires some manual configuration. With the assistance provided in this document, the configuration should prove relatively straightforward. Prior to enabling CountryCensor within a rule, you must configure it. All configuration takes place within CountryCensor.xml. There are two options for adding countries to be checked by CountryCensor: Adding the two-letter country code for a specific country Adding a META group, which includes all countries that reside within that region Two-letter country codes and their corresponding countries for use within CountryCensor are listed at the bottom of CountryCensor.xml. The countries included in each region and their corresponding groups are listed in CountryCensorGroups.xml. These files include many comments, and most of the options available are described in the files. To prepare CountryCensor to be used within a rule: 1. Launch a text editor (such as Notepad). 2. Edit CountryCensor.xml 3. Within the file, locate the Group entitled BlacklistedCountryCodes and add the desired two-letter country codes as seen below: If you wish to include all of the countries within a region, add the region here as well, see below: 17

18 By default, BlacklistedCountryCodes includes a META group named TopSpammers, which is simply a group including the top thirteen spam-producing companies. This group is merely provided as a demonstration of how to create and use META groups. You may choose not to use it. In any case, it should not be used until it has been modified to include or exclude whichever countries are suitable for your environment. You can safely add or remove two-letter country codes from this group and include it in your CountryCensor check by adding it to BlacklistedCountryCodes as shown below: IP addresses of servers that should be excluded from CountryCensor checks should be added to the section entitled CCBlacklistExclusions. Each IP address should be on a line by itself. 4. Once satisfied with the configuration options, save the file and close the text editor. When you have finished editing the configuration file, you can use CountryCensor within a rule. The sample settings illustrated above will cause CountryCensor to trigger on the following countries: United States United Kingdom New Zealand Australia North America including the following: (AG,AN,BB,BM,BS,CA,CR,CU,DM,DO,GD,GP,GT,HN,HT,JM,KY,LC,MQ,MX,NI,PA,PR,PY,SV,TT,US,VG,VI) 18

19 TopSpammers including the following by default (updated in release 6.1.8): (US,CN,ES,KR,FR,PL,BR,DE,RU,IN,IL,IT,GB) Using CountryCensor within MailMarshal SMTP The steps described in this section enable CountryCensor to trigger on a message that has traversed servers in any of the countries defined in CountryCensor.xml. To enable CountryCensor: 1. Start the Rule Wizard by right-clicking an existing Policy Group and selecting New Rule. 2. Select Next until you arrive at the Rule Conditions window and select the checkbox Where message is categorized as. 3. Create a rule that uses CountryCensor.xml. You will see a window as below: You should finish with a rule as seen below: Standard Rule: Block Spam CountryCensor Banned Countries When a message arrives Where message is incoming Where message is categorized as 'CountryCensor' Move the message to 'Banned Countries 19

20 Note: You should use this rule in conjunction with a comprehensive whitelist. While CountryCensor is a very powerful utility when configured properly, it makes no distinction between legitimate and unsolicited mail from a blocked country. An extensive variety of options and tricks can be used with the CountryCensor technology. You could decide to list (and block) a few countries that are known to be major spam producers. You could use it to define a list of allowed countries, then quarantining from all but the known friendly countries. HELO Rules New in MailMarshal SMTP 2006 is the ability to reject a message based on the validity of the connecting SMTP server. Spammers will frequently attempt to send your own IP address as their HELO name in an attempt to fool some older filtering systems. Per RFC specifications, a HELO name should be a server s fully qualified domain name as published in DNS. It should also match the connecting system s PTR record. MailMarshal SMTP now has the ability to reject a message solely based upon the HELO name used in the initial SMTP handshake. Creating this new type of rule simply requires creating a new Receiver rule. A typical HELO rule would look as follows: All of the typical Receiver rule options still apply. The options available for checking the HELO name are seen below: 20

21 Previously this type of connection error would have been filtered out with a category script. With the release of MailMarshal SMTP 2006 however, MailMarshal SMTP allows the messages to be rejected before wasting the bandwidth and CPU cycles of receiving and scanning the message headers and body. Note: Use this condition with caution. As with other Receiver rules, it causes to be rejected permanently with no further notice. TextCensor Scripts The easiest configurable addition to the default rules enabled within MailMarshal SMTP is the creation and use of TextCensor scripts within the existing rules. The MailMarshal Configurator provides a simple graphical interface for creating and modifying TextCensor scripts. By default MailMarshal SMTP includes a TextCensor script entitled Administrator maintained keyword list. If a rule is enabled to utilize this TextCensor script, an administrator simply needs to update the referenced TextCensor script. The changes to the configuration must then be committed before they will take effect. This allows the administrator to make immediate updates as they see spam messages missed by the current set of checks. The next section will cover the creation of this rule, as well as the various options available for its use within TextCensor scripts. Using TextCensor Scripts within Rules Once a TextCensor script has been created, it will then need to be referenced within a rule in order for its checks to be measured against messages. For example, if the Block Specific Spam rule is currently not created, the following steps can be taken to utilize the script: 1. Start the Rule Wizard by right-clicking an existing Policy Group and selecting New Rule. 2. Create a new Standard rule that reads as follows: When a message arrives Where message is incoming Where message triggers text censor script(s) Spam - Administrator Maintained Keyword list Move message to Spam 21

22 With this set up, when new spam variants come through that are not yet picked up by MailMarshal SMTP s definitions, administrators can simply add new entries to the referenced TextCensor script. This in turn will block the new spam variants before they become an issue. TextCensor Creation Options To create a new TextCensor script, simply open the Configurator, expand policy elements, right-click TextCensor Scripts and select New. This window allows you to import a Script from a file, export the displayed script to a file, or add new checks to the TextCensor script. To add a new word or phrase to the TextCensor script, first select which parts of the message the script should apply to. Then add items to the word weighting and matching list by clicking New. You can test your work by clicking Test. When you have finished entering values, click Sort. Then click OK. To edit items in the word weighting and matching list, select it and click Edit. Delete an item by selecting it and clicking Delete. When you have finished editing values, click Sort. Then click OK. To locate a word or operator in the matching list, click Find. Then enter the text to search for and search options. Click Find Next to search. The fields on this window are defined as follows: 22

23 Script Name Specifies a name MailMarshal SMTP will use to identify this script in rules and action logs. Apply this script to the following parts of a message Specifies the parts of an message MailMarshal SMTP will evaluate using the script. The available parts are: Message Header Message Subject Message Body Message Attachments Note: The script will be applied separately to each part of a message. For instance, if both headers and message body are selected for evaluation, the script will be evaluated once for the headers, then again for the body. Script triggering is not cumulative over parts. Enable matching of special characters This specifies any non-alphanumeric characters that TextCensor should treat as text. By default only alphanumeric characters can be entered into TextCensor items. Select the checkbox to enable matching for special characters and enter any you need to match. For instance, to match the HTML tag fragment "<script" you must enter the < in this field. Word weighting and matching List This specifies a list of TextCensor items that make up the TextCensor script. Change the items in this list using the New, Edit, and Delete buttons. The columns in the list are defined as follows: Weight - The base weighting value for this item. Type - The weighting type for this item, it specifies how TextCensor will count multiple matches of the item. Words and Phrases - The body of the item. Weighting trigger level - Specifies the total of triggered item values that will make the script "trigger" or return true. There are also a variety of keywords available for use within TextCensor scripts which allow not only the word but the context of the word to be evaluated. The following are the available key words: AND - Specifies that one word or phrase AND another word or phrase must appear within the text Usage: pills AND enlarge Behavior: triggers if both pills and enlarge are contained within the section specified for scanning OR - Specifies that one word or phrase OR another word or phrase must appear within the text Usage: cash OR c@sh Behavior: triggers if either cash or c@sh are contained within the section specified for scanning FOLLOWEDBY - Requires that the word or phrase after the FOLLOWEDBY keyword, comes immediately after the word or phrase. This can be further refined to check for the second keyword s presence within a set number of words distance from the first keyword. Usage: reduce FOLLOWEDBY debt Behavior: looks for debt following within 5 words (by default) of reduce in the section specified for scanning 23

24 Usage: home FOLLOWEDBY=3 loan Behavior: looks for the presence of loan up to three words after home NEAR - Requires a (NEAR) keyword of phrase comes either before or after another (NEAR) key word or phrase. This can be further refined to check for a word s presence within a set number of words distance from another word. Usage: pharmacy NEAR online Behavior: checks for the word online within 5 words (default) of pharmacy, not taking into account which one comes first. It could be pharmacy online or online pharmacy, the NEAR operator will trigger either way. Usage: pharmacy NEAR=8 online Behavior: checks for the presence of pharmacy within 8 words of online NOT - The check will trigger if the words specified are NOT present. Can also be used in conjunction with NEAR or FOLLOWEDBY to invert the sense of the test it is attached to. Usage: NOT marshal.com Behavior: This check will trigger if the section specified to be scanned does not contain marshal.com Usage: a FOLLOWEDBY=20 NOT b Behavior: matches string "a" that is not within 20 words of string "b" INSTANCES - Triggers if the expression appears a certain number of times within the text being scanned. Usage: money INSTANCES=3 Behavior: triggers if the word money appears 3 times within the text All of these keywords can be combined with other keywords to further refine the exact text that the script should trigger on. Examples: (see OR read) FOLLOWEDBY=3 full story ((joke OR word OR quote) FOLLOWEDBY=2 of the day) OR wotd OR jotd OR qotd (result of OR results OR draw OR draws OR winners) NEAR=20 (lottery OR lotto) (live FOLLOWEDBY=2 feeds) AND NOT satellite Though not as flexible as category scripts, TextCensor scripts are an effective means for performing a multitude of checks. They can simply search for specific words or the presence of a word in an unfamiliar context. The ability to create and modify TextCensor scripts enables the administrator to manage custom filters via a simple graphical interface. Note: TextCensor Boolean combinations do not span across the components of a message. pills AND enlarge will not trigger if pills is found in the subject and enlarge is found in the message body. NOT marshal.com will trigger if this phrase is not found in each selected part of the . Attack Prevention MailMarshal SMTP provides settings that allow you to protect your system against Denial of Service attacks (DoS) and Directory Harvest Attacks (DHA). DHA attacks in particular are used by spammers to determine valid addresses at your domain. Directory Harvest Attack (DHA) Protection When enabled, DHA prevention guards your system against Directory Harvest Attacks (DHA). MailMarshal SMTP s DHA protection can detect a DHA, drop the connection from the connecting server and blacklist the server for a specified length of time. 24

25 MailMarshal SMTP recognizes an attack when a remote server sends many messages to invalid users. Before enabling this feature, you must provide MailMarshal SMTP with a list of valid users. The easiest way to populate such a list is to import users from your mail server or Active Directory, using an LDAP or AD connector. See the section Whitelisting Practices, below, for more details. Setting up DHA Protection Setup for this feature is accessed through the Configurator under Tools > Server and Array Properties > Attack Prevention. Setup options and requirements differ slightly depending on the release of MailMarshal SMTP 2006 that is installed. Important Note: Before using DHA Prevention, you must provide MailMarshal SMTP with a list of all valid addresses within your organization. MailMarshal SMTP releases and earlier use a group entitled All Employees for this list. The All Employees group should NOT be renamed, nor should it be deleted. To use other groups, insert them into this group. MailMarshal SMTP release and above allow you to select one or more groups that contain the list of valid users. For details of the setup requirements for this function, please review the User Guide and Help for your installed version of MailMarshal SMTP. Spam Management Tips There are many different ways to handle messages once MailMarshal SMTP has identified them as spam. Header Rewriting MailMarshal SMTP has built-in header matching and rewriting ability. This feature can be used to tag the header to flag the message as spam. Then, instead of quarantining the message, it can be passed through to the end-user client where automatic rules can determine what to do with it. The message may, for example, be automatically moved to a Possible Spam folder for the end user to periodically review at their convenience. The following header rewriting configuration tags the subject line with [SPAM]. This rule is included within the default MailMarshal SMTP 2006 rules. If upgrading from an earlier version of MailMarshal SMTP, this can be used in a rule as follows: Standard Rule: Modify Subject Line of Spam When a message arrives Where message is incoming 25

26 Where message is categorized as 'Spam' Rewrite message headers using 'Rename Spam Subject' And pass message to the next rule for processing. You are not limited to rewriting the subject line. MailMarshal SMTP can also be used to add custom headers. For example, you may want to add a custom header field called X-MailMarshal and add Spam in the field. This has the advantage of keeping the subject line intact and the end-user s client (depending on the type) can usually be configured to detect its presence. The rule is as follows: Standard Rule: SpamCensor Flag Suspected Spam When a message arrives Except where addressed from 'Friendly Listservers' Where message is categorized as 'Spam' Rewrite message headers using 'Add X-Marshal Header' And pass message to the next rule for processing. To configure the custom header go to the Rule Wizard. In the Rewrite Message Header action, add a custom field as illustrated below. There are standards relating to header fields so ensure your fields start with X- and use only alphanumeric characters, see below. The second step is to add an entry to the field, in this case Spam. This is illustrated below. The header field will look like this: X-Marshal:Spam 26

27 You should finish with a rule that looks like the following: These approaches place the responsibility for handling a detected spam message on the client. This has the added benefit of allowing users to manage their own junk mail. It also ensures the mail administrator isn t responsible for the incurrence of any false positives. This next option alleviates the need for an end-user to undertake any mail client configuration. Tip: Sometimes users want to know what SpamCensor rules a message triggered. The logging result of the SpamCensor can be appended to an message with a message stamp. It can also be added to a message notification template, using a MailMarshal SMTP variable: {SpamCensorResult} In either the message stamp or notification template, type a { character to view a list of available variables, and select SpamCensorResult. Quarantining Detected Spam Rather than relying on client configuration, MailMarshal SMTP has the ability to quarantine a message at the server side. This is before it reaches the end-user s inbox. It is also the default behavior for most of the existing Anti-Spam rules within MailMarshal SMTP. In order to move a message to a folder rather than flagging it, you simply need to navigate to the Rule Actions pane of the new rule wizard. Then select Move message to folder. It is typically easier to move all spam messages to the same folder and the reason for this will be discussed later on in the document. However a quarantine rule would look similar to the following: 27