Email Sender Authentication Ferris Research Analyzer Information Service May 2007. Report #713 Ferris Research, Inc. 408 Columbus Ave., Suite 3A San Francisco, Calif. 94133, USA Phone: +1 (415) 986-1414 Fax: +1 (415) 986-5994 www.ferris.com
Recent Reports From Ferris Research Four New Messaging Products and Services Achieving Regulatory Compliance With Email and Internet Content Security Policy Enforcement Key Messaging Issues: 2007 and Beyond Meeting the Challenge of Email Discovery Spam Control: The Current Landscape Planning and Implementing an Email Archiving Solution Instant Messaging: New Developments in Presence and Real Time Information Interchange Identum s Private Post: Innovation in Email Encryption Domino Unified Messaging Product Selection and Implementation Trends in Outbound Content Control Reputation Services and Spam Control Pushing the Limits on Exchange Storage The Total Cost of Ownership for Voltage Identity-Based Encryption Solutions Assessing and Managing the TCO of Mobile Messaging Devices Email Archiving Technology Trends Mobile Messaging for Exchange: Product Selection and Implementation Issues Snapshot: Lucid8 GOexchange Preventive Maintenance Microsoft's Latest Push for Notes and Domino Migration Exchange Reliability and Its Impact on Organizations Snapshot: Teneros Application Continuity Appliance for Microsoft Exchange Implementing Email Archiving The Benefits of Integrating Enterprise Content Management Systems and Team Workspaces Enterprise Mobile Messaging Survey The Email Archiving Market, 2006-2010 Exchange 12 Assessment Anti-Spam Technology in the Asia-Pacific Region Why Exchange 12 Will Be 64-Bit Only Top 10 Messaging & Collaboration Issues: 2006 The SyncML Standard and Its Impact on Mobile Messaging Snapshot: Azaleos OneServer Boundary Email Security: The First Line of Defense Oracle Content Services: An Alternative to SharePoint Services for Enterprise Content Management The Plan for AOL Instant Messaging 2007Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. 2 Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies.
Table of Contents Executive Summary...4 The Case for Sender Authentication...5 Authentication Protects Your Brand...5 Reputation Services...6 The Problem With DNSBLs...6 Authentication Enables Domain-Based Reputation...6 How It Works...8 SPF/SIDF...8 Problems With SPF...9 DKIM...10 Problems With DKIM...10 SPF vs. DKIM...10 Using Authentication...11 For Senders...11 Audit the List of Legitimate Senders...11 Maintain Your Understanding...11 Use Both Standards...11 For Legitimate Bulk Senders...11 For Recipients...12 Standards Status...12 SPF...12 DKIM...12 Other Resources...13 Contributors to Authentication...13 SPF...13 DKIM...13 Useful Web Links...13 For More Information...14 2007 Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies. 3
Executive Summary Email sender authentication (sometimes called authorization ) is a way of detecting forgeries. It allows Internet domain owners to specify rules so that recipients can determine whether or not an incoming message is from the purported sender. It s very easy to forge the sender identity of an Internet email message. However, most forgery techniques can be detected by the recipient s message transfer agent (MTA), if the legitimate sender has published its authentication credentials. Using these credentials, recipients can tell legitimate messages from forged ones. This report looks at the role of authentication in blocking spam. It describes the two leading methods: SPF/SIDF (Sender Policy Framework/Sender ID Framework) and DKIM (DomainKeys Identified Mail). Finally, it provides steps that senders can take to protect their brand s reputation from forgery, and the steps that recipients can take to protect themselves from incoming spam and emailed viruses. Key findings in this report include: The credibility of email for conducting business transactions is at risk, due to imperfections in spam control technologies. There s no easy way for recipients to tell whether an email message is forged. This problem is damaging consumer trust in email. Reputation services are a newer approach to rating senders based on the actual email sending behavior. While reputation services can track reputation by IP address, they can t track reputation by domain, unless forged email addresses can be detected. Authentication is a foundation for building this domain-based sender reputation. Authentication does not filter spam, but it can improve the accuracy of a spam filter. This is why it is increasingly employed as a defense against phishing. 2007Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. 4 Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies.
The Case for Sender Authentication Email forgery is rampant today. But back in the early days of Internet email the early to mid-1980s the Internet was populated almost exclusively by researchers and academics. They generally did not foresee the commercialization of the Internet, nor the rise of Internet crime. So Internet email was originally designed to be somewhat flexible, without rigid security. Unfortunately, spammers, phishers, and virus writers have abused this openness. This open design allows these bad actors to send forged email. Email sender authentication allows an email recipient to detect such forgeries. It allows Internet domain owners to specify rules so that recipients can determine whether or not an incoming message is from the purported sender. Authentication Protects Your Brand Internet domain names e.g., paypal.com, whitehouse.gov, ferris.com are valuable assets. This is illustrated by the thriving market in domain names. Increasingly, an organization s domain name and its brand are interchangeable or indistinguishable. Therefore, forgery of email addresses can damage the company s brand equity. There s no way for recipients to tell by looking at the From line whether it has been forged, unless they are technical experts. That s why the vast majority of phishing email messages are forged. Authentication can improve people s confidence in email. Today, email s status as a medium for conducting business transactions is at risk, due to the inadequacy of most anti-spam technologies. Customer confidence has been undermined by the significant volumes of phishing and fraud-related spam that get through their spam filters as well as by false positives when legitimate and often important emails are blocked as spam. Authentication also improves the reliability of email delivery, by enhancing spam control technology. Further, it can protect the credibility of a brand, by detecting those attempts to forge it in email. 2007 Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies. 5
Reputation Services Authentication provides something like a license plate for the sending domain. License plates tell the authorities and other drivers who you are but not how well you drive. However, once we know who you are, the authorities can keep track of your driving record and punish you for infractions. Without authentication, email is like a highway where the cars have no license plates. If a red pickup truck runs a stop light, it would be futile and unfair to punish all red pickup trucks. Likewise, if you drive a red pickup truck, there s no way you can prove you re not the bad driver. The license plate itself doesn t stop people from driving poorly, just as authentication can t stop someone from spamming. But because of license plates, we re all motivated to drive more responsibly, and the authorities can tell the good drivers from the bad. In the same way, we can track the reputation of sending domains using authentication. The Problem With DNSBLs For several years, spam and virus control has been assisted by the use of Domain Name System (DNS) blacklists (DNSBLs). These lists compile rogue IP addresses and address ranges that have been observed sending spam, viruses, or other undesirable content. The lists are interrogated in real time, usually via a DNS query. Some DNSBLs gained the reputation of being run by amateurs who carelessly blacklisted legitimate senders of bulk email. While not all DNSBLs are badly run, there have been several high-profile examples of DNSBL errors that have caused a significant number of false positives. Several spam control vendors use a form of DNSBL known as a reputation service. These reputation services provide a professionally run service that rates the reputations of IP addresses good, bad, or unknown. Authentication Enables Domain-Based Reputation Authentication is a foundation for building domain-based sender reputation (i.e., track the email behavior of a domain). Today, we have IP address-based reputation services, but not the ability to track and report the reputation of a sending domain. This would be useful when: Several domains share an IP address. Email from a domain comes from several different IP addresses. Tracking a domain name, rather than an IP address, would make it possible to see all of the email sent from a domain, regardless of the computer. 2007Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. 6 Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies.
In the future, reputation services will be able to track the reputation of sending domains as well as IP addresses. This is not possible today, as the purported sender of a message is too easy to forge. Reputation services can t accurately track the reputation of a sending domain unless forged messages can also be detected. Authentication thus provides the missing piece of the puzzle, by allowing services to track the reputation of a domain. So, as the use of sender authentication becomes more widespread, reputation services will become more useful. In the future, they will be able to speak to the reputation of the sending domain, not just the particular IP address. 2007 Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies. 7
How It Works There are several schemes for authentication. This section compares and contrasts the two most prevalent. SPF/SIDF The main sender authentication technologies used today are SPF and its Microsoft-extended cousin, SIDF. SPF/SIDF allow domain owners to publish a list of IP addresses that are authorized senders of email for the domain. For simplicity, we ll just write SPF below, unless we need to emphasize the duality of the technologies. Figure 1 illustrates how an incoming message is checked using SPF. FIGURE 1 HOW SPF WORKS SPF SPF examines the SPF record of the purported sender s domain, comparing it with the sending IP address. Sources: Microsoft and Ferris Research. 1. The sending MTA transmits email. 2. The receiving MTA receives email. 3. The receiving MTA validates by comparing the purported sender s domain and the sending IP against the sender s SPF record. 4. If it passes, the receiving system looks up the domain s reputation. 5. The receiving system determines the final disposition of the message. 2007Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. 8 Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies.
A domain owner who wants to participate in SPF-style authentication will publish a list of IP addresses or IP address ranges in the DNS, the Internet s white pages. This list known as an SPF record tells potential recipients which IP addresses are authorized to send on behalf of this domain. Note that a message may have more than one purported responsible domain. Such domains may appear in any of the following SMTP commands or parts of the email message: Envelope sender (i.e., the MAIL FROM command parameter, usually reproduced in the Return-Path header) HELO or EHLO command parameter From header Sender header Resent-From header Resent-Sender header In most messages, all the domains in these parts will be the same, where present. Different implementations of SPF and SIDF will use different rules to choose the domain on which it bases its decision: The classic implementation of SPF tests the envelope sender domain (as presented in the MAIL FROM or HELO/EHLO transactions). Microsoft s original Sender ID implementation only tests one domain selected from the other four headers in the above list (the algorithm for selecting which header to use is known as Purported Responsible Address, or PRA). Problems With SPF SPF can be confused by email forwarding (strictly, redirection), such as is done for university alumni who want to maintain a university email address, or for holders of vanity domains that provide email forwarding. An automatically forwarded email can cause the receiving MTA to see the IP address of the forwarding MTA but compare it with the SPF record of the original sender. This can cause the MTA to flag the message as a forgery. The problem happens when the message is forwarded but the sender doesn t change a common occurrence today. There are ways around this problem, but they mainly rely on the manager of the forwarder to install new versions of the MTAs that write additional headers to the message, indicating the original sender. 2007 Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies. 9
DKIM DKIM is a little more complex than SPF. It uses digital signatures to detect forgery. The signatures are not used end-to-end as in schemes such as S/MIME but MTA-to-MTA. In other words, there s no impact on the users email client software. A sending domain has a public/private key pair. The domain owner publishes the public key in the DNS. The sending MTA uses the private key to sign the message including some of the message headers. The receiving MTA retrieves the public key from the DNS and verifies the signature using the public key. (Note that there s no need for an expensive certificate authority; domain owners can create their own key pair using OpenSSL.) A domain owner also has a good measure of flexibility in setting up DKIM. For example, the owner can delegate keys to third parties for special purposes or revoke old keys. As one might expect, DKIM is more complex to set up, but it s worth the extra effort because of the additional robustness. Problems With DKIM DKIM can become confused if an MTA modifies the message. This is because the digital signature will no longer match the modified message. This might happen if an MTA: Adds a custom disclaimer footer after the point of signing. Rewrites or reorders message headers this can happen with some badly written mailing list software. Messaging managers should ideally ensure that messages they handle aren t willfully modified once the signature has been generated. SPF vs. DKIM To extend the license plate analogy: If SPF is like a license plate, then DKIM is like an E-ZPass more sophisticated, but also more complex. However, it s not a case of either SPF/SIDF or DKIM. The two technologies are complementary. Ideally, sending domain owners should use both technologies, and receiving spam control solutions should use both in their suite of spam tests. 2007Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. 10 Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies.
Using Authentication For Senders Audit the List of Legitimate Senders As a domain owner, you need to audit your organization s use of the domain in order to determine which IP addresses are used for sending email. You can then include these IP addresses in the SPF record and set up the correct DKIM key pairs. Building a complete list of such senders is often a time-consuming task. One complication is that it s common for different departments in an organization to contract with third parties to send their email such as direct marketing services. In this case, the external IP address belongs to the contractor, even though the sender s domain is listed as the primary address. You should discover who the email-sending stakeholders are such as marketing, e commerce, and customer relations and ensure that the management chain buys into the effort. It should be in their interests to participate in authentication, as it will increase the deliverability of their email. Maintain Your Understanding You should also ensure that you regularly update your list of IP addresses. As the list changes, update the SPF record and make any necessary changes to the DKIM key pairs. Failure to react to changes will affect the deliverability of the organization s email. Conversely, too broad an SPF record will open the domain up to increased risk of spoofing. The extreme case is the SPF record that allows any IP address to send on behalf of the domain in that case, receiving MTAs won t be able to detect a forgery. Use Both Standards Some recipients may only check against SPF or DKIM, so you should ideally use both. Using both will produce better results better deliverability and fewer instances of forgery. For Legitimate Bulk Senders Using authentication will improve deliverability, especially for legitimate direct marketers or other bulk senders. Authentication is increasingly important. 2007 Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies. 11
For example: It s often required before you can subscribe to ISPs anti-spam feedback loops. For example, AOL s feedback loop (FBL) service will forward to you any email from your IP addresses that are reported as spam by AOL members. Yahoo Mail displays positive reinforcement if a message passes a DKIM forgery check. Windows Live Mail (Hotmail) displays a warning banner if a message fails an SIDF check. For Recipients The provider of your spam control product or service should incorporate SPF and DKIM forgery tests in its cocktail of spam control techniques. In the future, your vendor should add domainbased reputation checks into the mix. Note: We don t recommend that forgery tests be the only reason to reject an incoming message. In most cases, it s far too aggressive to reject messages that fail authentication. Authentication should be just one of a battery of tests that your spam control filter employs. In the future, your vendor should add domain-based reputation checks into the mix. Standards Status Authentication is a usable technology today. The accuracy and usefulness of authentication will continue to increase as more domains participate, as reputation services get smarter, and as the standardization process advances. SPF Thanks to some intellectual property disagreements within the Internet Engineering Task Force (IETF), both SPF and SIDF failed to progress toward a formal standard. SPF/SIDF are now officially designated as Experimental RFC 4408. Although not a de jure standard, SPF-Classic is now essentially a de facto standard. As of April 2007, there were about seven million domains with SPF records, worldwide. DKIM DKIM will soon be an IETF standard (i.e., a published RFC). Draft 10 is expected to be published by mid-2007, after a final editing process. DKIM is already widely deployed and is backward compatible with a predecessor proposal, known as DomainKeys. 2007Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. 12 Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies.
Other Resources Contributors to Authentication Here is a short list of some major contributions to the current state of the art. It is not intended to be exhaustive. SPF SPF is the latest in a long line of proposals and research, aimed at lightweight authentication of incoming SMTP connections and based on lists of authorized IP addresses. In 2002, Paul Vixie wrote a short paper, Repudiating Mail-From. Key predecessors also included Hadmut Danisch s Reverse MX (RMX) and Gordon Fecyk s Designated Mailer Protocol (DMP). In 2003, Meng Weng Wong merged the RMX and DMP specifications to form the original Sender Permitted From (SPF) proposal later renamed Sender Policy Framework. Microsoft developed a similar scheme, called Caller ID for Email, which included the PRA algorithm (see the earlier section, How It Works ). DKIM In 2004, Yahoo proposed DomainKeys. At roughly the same time, Cisco developed a very similar proposal, known as Identified Internet Mail. In 2005, the Mutual Internet Practices Association worked to merge the proposals, to create the first version of DKIM. Participants included Alt-N, AOL, Brandenburg InternetWorking, Cisco, EarthLink, IBM, Microsoft, PGP, Sendmail, StrongMail, Tumbleweed, VeriSign, and Yahoo. Useful Web Links http://www.openspf.org/ http://www.microsoft.com/senderid http://www.dkim.org http://tools.ietf.org/wg/dkim Author: Richi Jennings Editor: Sue Hildreth 2007 Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies. 13
For More Information This report is based on the Ferris Research Sender Authentication webinar held February 21, 2007. The presentation slides and an audio recording of the webinar are available for download by subscribers to our Analyzer Information Service at www.ferris.com. They are available for purchase by nonsubscribers as well. The webinar was moderated by Ferris analyst Richi Jennings. Ferris Research would like to thank the following presenters: Eric Allman, Co-founder and Chief Science Officer, Sendmail Joshua Baer, CTO, Datran Media Harry Katz, Program Manager with the Technology Care and Safety Group, Microsoft 2007Ferris Research, Inc. All rights reserved. Not to be reproduced without this notice. 14 Visit us at www.ferris.com for market intelligence on messaging and collaboration technologies.
Ferris Research Ferris Research is a market research firm specializing in messaging and collaborative technologies. We provide business, market, and technical intelligence to vendors and corporate IT managers worldwide with analysts located in North America, Europe, and the Asia-Pacific region. To help clients track the technology and spot important developments, Ferris publishes reports, white papers, bulletins, and a news wire; organizes conferences and surveys; and provides customized consulting. In business since 1991, we enjoy an international reputation as the leading firm in our field, and have by far the largest and most experienced research team covering messaging and collaboration. Ferris Research is located at 408 Columbus Ave., Suite 3A, San Francisco, Calif. 94133, USA. For more information, visit www.ferris.com or call +1 (415) 986-1414. Free News Service Ferris Research publishes a free daily news service. It provides comprehensive coverage of the messaging and collaboration field, and is a great way to keep current. Topics include spam, email, email retention/archiving, mobile messaging devices, consumer messaging services, Web conferencing, email encryption, email migrations and upgrades, regulations compliance, instant messaging, ISP messaging, and team workspaces. The news is distributed daily. To register, go to www.ferris.com/forms/newsletter_signup.php. In addition, you will receive one or two emails every month announcing new Ferris reports or conferences. To opt out and suppress further email from Ferris Research, click on the opt-out button at the end of each news mailing.