ANTI-SPAM SOLUTIONS TECHNOLOGY REPORT FEBRUARY 2006 Email Systems Managed Service
2 ANTI-SPAM SOLUTIONS TECHNOLOGY REPORT Contents Email Systems Managed Service Test objectives and scenario...3 Test network...4 Test methodology...5 Product test reporting...6 Certification...7 The product...8 Test report...10 Test results...15 West Coast Labs conclusion...16 Security features buyers guide...17 West Coast Labs, William Knox House, Britannic Way, Llandarcy, Swansea, SA10 6EL, UK. Tel : +44 1792 324000, Fax : +44 1792 324001.
EMAIL SYSTEMS MANAGED SERVICE 3 Test objective and scenario The war for control of corporate inboxes has been raging for some years now as Anti-Spam solution providers seek to protect us from unsolicited, inappropriate and often offensive intrusions into our time. The originators of these emails are becoming ever more inventive and so more and more companies are coming to rely on automatic solutions with learning engines to protect their users and machines. The emails themselves are getting more sophisticated. Spam is now no longer just advertising material, but is evolving, and often acting as the precursor to identity theft. This Technology Report examines the functionality and performance of participating Anti-Spam products which are aimed at the small, midsize and corporate network environments. It has been open to both software and appliance-based solutions plus hosted services. The objective of our overall testing program, which is open to all Anti- Spam Vendors is, through a real-world test environment, to provide an independent validation of Anti-Spam solution effectiveness with particular reference to: A detailed view of the features and functions of the solutions Spam detection capability and rates of detection of each solution Integration into a network infrastructure and level of administration required to operate effectively.
4 ANTI-SPAM SOLUTIONS TECHNOLOGY REPORT Test network Software solutions are installed on servers that exceed the minimum specifications required by the vendor. Appliance-based solutions are installed on the network according to the vendor s recommended placing. For hosted services, WCL test through identified email accounts and will change the MX records to divert the mail stream through the hosted service. In order to allow the DNS change to propagate, service providers allow a 2-day settling-in period. Details of the tuning and vendor customer support will form part of the additional feature testing and reporting.
EMAIL SYSTEMS MANAGED SERVICE 5 Test methodology WCL has a number of domains available which act as honeypots for spam, receiving genuine, not canned spam. These domains receive varying levels of spam and are intended to mirror different email environments. Within each domain are designated user accounts with a variety of email practices and needs - some are subscribed to a variety of newsgroups and mailing lists. Some user accounts actively contribute to mailing lists. The domain designated for testing purposes will be that which currently receives spam at a level consistent with the test requirements. For testing in this Technology Report and for the certification of each of the participating solutions, we used live mail feeds coming in to various extra domains wholly owned and controlled by West Coast Labs. Each domain used contains a number of individual user accounts with established email addresses, along with distribution lists. To maintain the flow of genuine mail, test engineers used several internal and external accounts, to send emails that simulated real life email transactions common in business: for example requesting meetings, sending notifications to groups and non-business related social emails. Emails were also sent from web-based accounts to simulate external users sending non business-related emails and home workers. Individual user accounts were subscribed to several mailing lists and daily newsletters for grey mail purposes. For each solution we configured the device or software to fit in with the test network and placed it into a stream of live mail to see how it would cope in an out-of-the-box configuration with real-world traffic. However, we do recognize that a large part of spam detection relies on an initially intensive learning process. Hence, we will be placing these devices in the mail feed in coming months for longer periods of time, interactively training them, and updating the performance data included in the online White Papers.
6 ANTI-SPAM SOLUTIONS TECHNOLOGY REPORT Product test reporting For each product that we test, we will issue a report which will address the following aspects of the product: 1. Management/Administration Ease of Setup/Installation Ease of Use Logging and reporting function Rule creation Customization Content Categories Technical Support Available Program Help Menu 2. Functionality Email Processing Steps Allow/Blocking of Email Quarantine Area Additional functionality reporting Block Email Addresses Blacklist/Whitelist Allow Email Addresses 3. Performance Volume or % of spam detected False positive rate Spam incorrectly passed through Legitimate mail blocked Legitimate subscription mail blocked
EMAIL SYSTEMS MANAGED SERVICE 7 Certification - Checkmark Upon successful completion of the catch rate testing, participating solutions will be accredited to Checkmark Certifications for Anti-Spam subject to achieving the following catch rates:- Checkmark Anti-Spam Certification PREMIUM 97% and over Catch Rate. www.check-mark.com Checkmark Anti-Spam Certification STANDARD 90% and over Catch Rate. www.check-mark.com
8 ANTI-SPAM SOLUTIONS TECHNOLOGY REPORT The product Email Systems offers best-of-breed email protection, management and compliance services which collectively ensure that email is always available as a productive business tool for its customers in the SME and enterprise markets. url : www.emailsystems.com Email Systems says about the Business Benefit of its Managed Service Email Systems suite of services collectively free up the corporate resources bandwidth and user time - which spam would otherwise consume, whilst offering protection from Denial of Service (DoS) attacks and reducing the risk of email failure for the business user. Powerful yet flexible rules engines allow organisations to monitor email traffic and prioritise business critical email, such as sales-leads. Image filters protect email users from pornographic material, thereby protecting employers from potential law suits. Email archiving features rapid search and retrieval which ensures that copies of all emails are always available, whether for disaster recovery, compliance or legal reasons. url : www.emailsystems.com
EMAIL SYSTEMS MANAGED SERVICE 9 The product Email Systems says about the Technical Benefits of its Managed Service Manages flow of email to corporate servers using a suite of multi-layered/ multi-technology approaches to remove spam and eliminate all viruses. Provides email buffering service if an organisation loses internet access or suffers internal server outage. Outbound email monitoring and filtering reduces risk of confidential documentation leakage, and protects company reputation. Mirrored infrastructure in different geographical locations, coupled with real-time replication minimizes risk. Comprehensive user management enables delegated management and monitoring. Real time audit trail of every email that is sent / received enables 100% trackability and accountability. Detailed graphical reporting for up to 3 years contributes to capacity planning. url : www.emailsystems.com
10 ANTI-SPAM SOLUTIONS TECHNOLOGY REPORT Test report Introduction Email Systems is a UK based company offering a managed service that corporations can route mail through before it gets to their servers. In addition to providing the anti-spam service tested here they also provide anti-virus scanning with the choice of Sophos or F-Secure. This approach removes the need for purchasing and housing extra hardware in house and also outsources the responsibility for protection.
EMAIL SYSTEMS MANAGED SERVICE 11 Test report Installation and Configuration The set up and configuration is very simple and this is helped by a comprehensive and easy to understand Service Deployment Plan. This sets out an eight stage process for administrators to follow that acts as a guide to switching the flow of a mail feed. Both the document and the process have been well thought out and have been written so that they can be used by either a novice or experienced administrator. Checklists are also provided throughout to ensure that all bases have been covered and that the change over is smooth. EmailSystems support also perform a scan of an organisation s current MX records to establish the current mail route and then provide a draft email pre-filled with the changes that need to be made and the contact email address for the hosting company. Once the changes have been made to the relevant MX records, mail will start flowing through the Email Systems service and the protection activates. Their support team is also very on the ball on the one occasion we had to contact them the response and resolution of our query was very speedy.
12 ANTI-SPAM SOLUTIONS TECHNOLOGY REPORT Test report The Interface The web based customer management portal is decked out in very tasteful subdued blues and greys that are easy on the eye. A good proportion of the interface is Flash based and the animations on some of the pages add a bit of sparkle to the user experience. The initial page after logging in shows a pie chart that breaks down the traffic for a 28 day period into colour coded slices, and displays the data on a per day basis below in histogram format. Also included here is an overview of the status of the accounts, a list of notifications of upgrades and improvements from support, the network status of each domain and a list of top viruses received, top recipients within the domains, and top inbound sender domains. Having all of this data easily within reach makes getting a snapshot view of how the domains are performing very easy. The main menu is listed across the top of the screen with the options being Users, Accounts, Domains, Logs, Reports, and Help. The Users section allows a current administrator to add further administrators with varying scopes they can be set up to make changes to individual domains or to the entire account. There is also the ability to restrict a user to a specific IP address for logins. These are useful features which allow for the dissemination of responsibility if a company has several domains, whilst still allowing one or more administrators to have overall control of the EmailSystems account.
EMAIL SYSTEMS MANAGED SERVICE 13 Test report The Interface (continued) The next section is Accounts and this allows the user to change the email address for administrative notifications for the entire Email Systems account covering all domains, and also to set rules that will apply across all domains registered on a simple Condition Action basis. These include conditions based upon message size, attachment types, specific words in either the body or the subject, phrases in the mail headers and time restrictions. The Actions include allowing the message, allowing with a spam filter override, copy, redirect, delete with no notifications or quarantine. There is also a further set of options that allows the user to specify notification and logging actions. The Domains section allows control over individual domains at a more granular level. Recipients of administrative notifications may also be specified here on a per domain basis, rules can be added for certain domains and not others in the same fashion described above, different mail signatures can be specified for each domain, and there are sections to configure alerts and filter settings.
14 ANTI-SPAM SOLUTIONS TECHNOLOGY REPORT Test report The Interface (continued) The Logs section provides a variety of different logs ranging from Mail in, Virus and Spam through to Quarantine and an Event Log. The logs are well formatted and cleanly presented, with the data easy to see. An administrator can quickly look at a message subject, recipients, sender and status to see which rules have been triggered for certain groups of emails. There are also good filtering options here to allow limitation to specific recipients, senders, subjects or date ranges. Messages in Quarantine can be viewed in their own window. This window displays the headers and then has a separate section for the message content, with the option to spawn the content in its own distinct window to examine the contents more closely if required. The Reporting section gives a set of overall reports and allows searches to be performed and then saved as favourites to be run again at a later date. These are presented as Flash animations and are in keeping with the look and feel of the rest of the interface. There are a large number of options available within the search parameters various different report types, timescales, comparative or aggregate reports over all domains, and direction of mail are all selectable and there are a variety of presentation options. Each scan is then saved under the Historic section so that it can be re-run later if needed.
EMAIL SYSTEMS MANAGED SERVICE 15 Results Type of Mail Delivered as Genuine (%) Delivered as Spam (%) GENUINE 100 0 SPAM 2 98 EmailSystems Managed Service performed well, delivering 100% of the genuine mail correctly and correctly classifying 98% of the spam mail in a straight out of the box configuration. It is also worth noting that the EmailSystems Managed Service solution delivers a good proportion of grey and list mail as genuine whilst still stopping a lot of the more questionable material. This gives an organisation the flexibility and opportunity to define policies during a training period without missing mail that could be potentially business critical.
16 ANTI-SPAM SOLUTIONS TECHNOLOGY REPORT West Coast Labs conclusion Overall EmailSystems Managed Service is a well presented and clean looking system with a lot of functionality. The options are easy to locate and the online help is well written and clear. It can be highly recommended to any company that wishes to take some pressure from a busy administrator or who wishes to outsource the responsibility for spam and virus scanning their mail to a third party The EmailSystems Managed Service software performed consistently well in the tests, and therefore West Coast Labs is pleased to award the EmailSystems Managed Service the Premium level Anti-Spam Checkmark. West Coast Labs, William Knox House, Britannic Way, Llandarcy, Swansea, SA10 6EL, UK. Tel : +44 1792 324000, Fax : +44 1792 324001.
EMAIL SYSTEMS MANAGED SERVICE 17 Security features buyers guide as stated by Email Systems SPAM FEATURES Does the product block spam out of box or does it require addition or tuning of rules? Blocks spam out of the box, a true managed service were Email Systems ensure the system blocks true spam. Is user feedback required over initial stage of deployment? Not required. FILTERING Does the product utilise keyword lists? Not for core spam filtering, but keywords can be used to block unwanted emails. Does the product utilise Bayesian filtering? Bayesian is used, but as a small element of the spam detection. Can white-lists/black-lists be set? As with keywords, they are avaliable but not to block spam / avoid false positives but to enhance user email experience... i.e. if a student wanted to study spam from hotmail addresses a white list could be used to only allow spam from hotmail through. Does product support RBL? YES. Does the product support the setting of different confidence levels? Can actions be varied at different confidence levels? YES, configurable per organisation, per domain or each user. Can actions be varied at different confidence levels? YES, block; tag; report only
18 ANTI-SPAM SOLUTIONS TECHNOLOGY REPORT Security features buyers guide as stated by Email Systems ADMINISTRATION Can the product be automatically updated? YES, managed service. Can filters be automatically updated? YES, managed service. What are the update methods? Automatic as central managed service, no user input required. Can suspected spam be quarantined? YES, all spam quarantined for 28 days as default and avaliable online via secure web interface. If so, what type of quarantine (forward to Q mailbox / saved on device / etc.)? Stored centrally on hosted system. Accessed by web broswer. END USER INTERACTION Can users see reports individual to them? YES. Can users process messages themselves? YES, via web administration portal Can users review mail marked as spam? YES, via web administration portal Can users free messages from quarantine? YES, via web administration portal Can users set their own white lists/black lists? Yes, via web administration portal
EMAIL SYSTEMS MANAGED SERVICE 19 Security features buyers guide as stated by Email Systems ADDITIONAL SECURITY FEATURES 1. Transport Layered Security (TLS) provides assurred encrypted email delivery 2. Encrypted web portal ensures privacy and protection of sensitive information 3. IP access monitor and blocking for web access 4. Pro-active DNS monitoring for client records ensures mail security loopholes can't be introduced 5. Multiple datacenters containing redundant hardware means no single point of failure 6. Unique replicated data solution ensures that there is no risk of loss of any e-mail messages 7. Lock down of access to web portal to designated IP ranges, further protects against hacking. 8. Anti-Virus solution using multiple best of breed vendors to provide enhanced protection 8. True "Zero Hour" Anti-Virus technology based on Real Time Detection, negating risks of early stage viruses evading traditional scanners. 9. Comprehensive image analysis tool set to protect employees & company reputation against exchange of pornographic material 10. 100% audit record of every message relayed by the service accessible by administrators and end-users if required. 11. End-user management system enables selected features to be accessed by end-users 12. Directory Harvest Protection ensures that company e-mail addresses are protected from capture and exploitation 13. Reputation Service contributes to spam detection and protects organizations from the risk of mass Distributed Denial of Service Attacks (DDoS).
20 ANTI-SPAM SOLUTIONS TECHNOLOGY REPORT Security features buyers guide as stated by Email Systems ADDITIONAL SECURITY FEATURES (continued) 14. High throughput, purpose built message switching technology further contibutes to the protection of extreme mail traffic brought about by DDoS attacks. 15. Comprehensive policy based content management system enables rules to be created to control who can exchange what content with whom 16. Detailed user management system, enabling access to be granted/ denied to functions within the system 17. Graphical reporting system collecting and reporting data for up to 3 years, supports investigations into policy violations or suspicious events. 18. Event log records policy violations, login attempts and changes made to the system enabling traceability and accountability. 19. Pro-active monitoring by 24 x 7 EMS Network Operations Centre ensures that any major e-mail incidents are indentified, managed and alerted to the customer before they can do damage. 20. Monitoring mode for certain policies enables non-invasive introduction of system. List up to 30 features url : www.emailsystems.com