How s are sent from Xero

Transcription

1 How s are sent from Xero Technical discussion In June 2013 we made a change to the way s are sent from Xero. Some of our users have asked us why the change was necessary and whether we are planning to provide more options in the future. Be warned this is a fairly technical topic so take a deep breath. Xero allows organisations to raise invoices and statements and send them to their customers. Sending on behalf of hundreds of thousands of Xero organisations, to the millions of their customers is a significant undertaking, and one that we take seriously. Recently we changed the mechanism by which we send customer invoice s. This post is an attempt to describe the new setup, the reasoning behind the decision, and a road map for future changes that we hope to make. Setting the scene: The landscape has changed dramatically over the past 5 years, and mail servers and ISPs have all developed unique practices to try and shield their users from spam and other unsolicited that currently floods the internet. Over the past 10 years, several standards and de facto practices have evolved to help mail servers make decisions on which s to trust: Sender Policy Framework (SPF) a 10 year old standard that lets the owner of a mail domain (e.g. gmail.com) specify which mail servers are allowed to send DomainKeys Identified Mail (DKIM) a slightly newer standard that allows the sender of an to digitally sign their to prove that it came from them Grey listing a heuristic that some receiving mail servers use to slow down the receipt of , on the theory that spammers won t attempt to queue and retry messages Spam filtering a set of algorithms that receiving mails servers can run on to try and determine if the content of the is spammy Blacklisting various databases of IP addresses or addresses that are known to send spam Domain based Message Authentication, Reporting & Conformance (DMARC) a recent specification that combines SPF and DKIM to allow senders to prove that the mail was sent by them We realise that our users absolutely, positively want that they send to be received by their customers their cashflow and business depend on timely payments. Even though is by design an unreliable communication mechanism, we are trying to make it as robust and reliable

2 as possible for Xero customers to use to send invoices to their customers. Our previous approach: Our previous approach to sending invoice s from within Xero was to impersonate or send on behalf of the user currently logged in to Xero. e.g. If had signed up to Xero, and proved that they controlled the address, then we would happily send out that looked like this: From: Joe Bloggs To: A Customer <a.customer@customer.com> Subject: Here is your invoice, pay me now! In 2006, this was a fairly reliable solution, and most mail servers would happily accept these s and deliver them to the recipient. In the subsequent years, it was possible for Joe Bloggs to add an SPF record to their flatnotfound.com to indicate that Xero was able to send on their behalf. This worked reasonably well for a few more years. The Problem: It is now no longer possible for SaaS vendors to reliably send on behalf of their customers. If a recipient mail server receives an like the above example, the mail server will perform some checks to try and decide whether this was legitimately sent by Joe Bloggs. One of the indications the server will use is the IP address of the server that delivered the message to it. If the server does not look like a legitimate server for Joe Bloggs ISP (flatnotfound.com), then that s a negative mark against the . Traditionally a combination of reverse DNS lookups and SPF would be used to perform this check. These days, setting SPF records is not sufficient to prove the authenticity of an , and this example illustrates the reason why: Envelope Sender: joe.bloggs@evilspammer.net From: Joe Bloggs <joe.bloggs@flatnotfound.com> To: A Customer <a.customer@customer.com> Subject: Here is your invoice, pay me now! The Envelope Sender address is never displayed to regular users, but it is actually the

3 address that SPF uses to test the authenticity of an . In this example, if this message arrived from a server that was legitimately run by evilspammer.net, then technically that s all that is needed to be legitimate from an SPF perspective even though the displayed in the recipient s mail client doesn t indicate it comes from evilspammer.net! For this reason, SPF is no longer the solution to sending on behalf of customers like it was when we launched Xero a few years ago. We have had this confirmed by experts from Facebook, Yahoo! and JPMorgan Chase when we spoke to them in February this year. In fact, they confirmed what we were seeing, which is that there s no way to reliably send while pretending to be someone else. The previous method of sending Xero s out by impersonating the Xero user was untenable, and we would see continuing increases in the numbers of s that would fail to deliver. Problems with SPF: We ve already discussed how SPF on it s own is not sufficient, and that it relies on the verification of a part of the message that s not actually displayed to an end user. Our invoice s actually looked roughly like this: Envelope Sender: xyzzy123@notifications.xero.com From: Joe Bloggs <joe.bloggs@flatnotfound.com> To: A Customer <a.customer@customer.com> Subject: Here is your invoice, pay me now! For this to pass an SPF test, the IP address of the mail server is compared to the SPF records of the Envelope Sender domain of notifications.xero.com. Xero has control of these records, and so we were able to make sure that the SPF record for notifications.xero.com was always valid. Some mail servers would also run SPF like tests against the visible From address that displays in the , even though that s not strictly part of the SPF standard, effectively asking the question: Is Xero s server allowed to send from flatnotfound.com? In order to pass this test, every Xero customer would need to create an SPF record in their DNS to allow Xero s mail servers to send on their behalf. For some Xero customers that are IT savvy, and in control of their own DNS, this was indeed possible, and over the past few years approximately one hundred Xero customers have managed to do this successfully.

4 However, for most Xero customers this is infeasible: Many Xero customers use an ISP address, rather than their own domain name (e.g. rather than This means that they have no control over the configuration of their ISP. Customers that had their own domain name (e.g. MyCompany.com) but weren t able to change DNS, had a large company with strict IT policies, or were part of a multi national. These customers aren t easily able to create SPF records. Customers that use SPF, but didn t include Xero s servers in their SPF records may be hard failing all impersonated mail from other sources. Our analysis showed this was potentially the case for about 7% of customers. Corporate customer s mail servers would often reject Xero invoice s that they sent to themselves, even if SPF was correctly set up. E.g. a corporate mail server for MyCompany.com would say I ve recieved mail from xero.com s mail server s that is pretending to be from MyCompany.com. I know that I m the only mail server for MyCompany.com, so this is obviously spam and throw it away. Creating valid SPF records is actually hard. One only needs to look around the internet and on our own Community to see that people don t understand the nuances of SPF, and a simple mistake in configuration can stop delivery of all your company s . This goes against our mantra of being the World s easiest accounting system. The (2013) solution: We absolutely want to deliver for our customers, to their customers. Asking our users to download invoices and send them from their desktop is cumbersome and inefficient although we realise for some customers that it will always be their preferred approach so that they have complete control. Candidate solutions: Continue impersonating Unreliable, not recommended by experts. Requires every Xero user to configure SPF correctly, even though that won t guarantee delivery. Send via our customer s servers Technically infeasible (we d have to collect login credentials, connect to hundreds of thousands of authenticated mail servers to send , and deal with the myriad of different ways that could fail). We d be outsourcing the delivery issues to our customers, rather than being responsible for delivering mail correctly on their behalf. Deliver with a.xero.com From address We can guarantee that messages are correctly authenticated. We control the full infrastructure for the sending.

5 We can track deliverability and bounces. Ultimately, we made the decision to choose the third option, and change the way that we deliver invoice . We did consider a hybrid approach, or allowing users to opt in to one or other approach, but we decided that there were benefits in having a single, reliable delivery mechanism that required no configuration by our customers. The change we made in June 2013 changed the format of our invoice s to look like this: Envelope Sender: xyzzy123@notifications.xero.com From: Joe Bloggs <message service@post.xero.com> Reply To: Joe.Bloggs@flatnotfound.com To: A Customer <a.customer@customer.com> Subject: Here is your invoice, pay me now! To break down the individual changes, and the rationale behind them: The Envelope Sender address, remains as a address. This is the domain that is used in SPF to validate that our mail server is allowed to send this . This address is not displayed in user s clients, and the unique part at the beginning of the address allows us to track bounces and failures back to the individual sender. The From address has been broken into two parts: The Display Name Joe Bloggs, which is now configurable per organisation in Xero The address message service@post.xero.com, which is no longer configurable The Reply To address is now configurable per organisation (e.g. Joe.Bloggs@flatnotfound.com). This is the address that any customer replies to your invoice s will be directed to and displayed to them, if they press the Reply button in their mail client. Our testing showed that in many mail clients, the recipient of this would see nothing untoward, and simple actions like reading and replying to s would not display the xero.com mail addresses. In some mail clients (e.g. Outlook 2013), the new format means that recipients can now see xero.com in the invoice s that you send. This is an unavoidable side effect of this technical decision, and it was not made for marketing reasons! The reason that some mail clients display xero.com, is that those clients want to show the user the information they have used to make the trust decision that has lead to the not being marked as spam. Our new format leverages xero.com s trust and delivery mechanism, so Outlook wants to make it clear to the user who receives the the reason why it has appeared in their inbox.

6 I ve labelled this as The (2013) Solution, because although this the best solution at the moment, the nature of the evolving war against spammers means that this approach is not guaranteed to work forever. Bounce messages There are several different types of replies that might happen to an . If a user is performing the reply, usually by clicking the Reply button in their mail client, then the reply will go to the Reply To address, and arrive directly in the original sender s inbox. If a user is using an old mail client that doesn t understand the Reply To address, or they manually reply to the From address of message service@post.xero.com, then their reply won t be sent to the original sender, but will rather be sent to the unmonitored mailbox hosted by Xero. We don t expect that this will happen too often, but in those situations that it does happen we want the experience for your customers to be understandable. So what we ve done is have a robot Auto Reply to all of those messages with an Oops! message asking them to re send their to you rather than Xero: This mailbox is unmonitored, and we don t read any of the s that are erroneously sent there. Some time after the auto reply is sent back to the original sender, the will get deleted from our servers. The other types of replies that will come from an invoice are sent automatically by the recipient s mail server, and called Bounces.

7 Bounce messages will usually happen in one of two ways: An immediate failure, which we ll send directly to the user (e.g. domain name doesn t exist, or has been mis typed) A slower failure, where the recipient mail server initially accepts the mail message but then later decides it won t deliver (e.g. the username doesn t exist, or their mailbox is full). We receive these bounce message to the Envelope Sender address of something@notifications.xero.com, and re route it back to the original sender at the Reply To address. Implementing the DMARC standard: Those of you that run your own mail servers will know how hard it is to track the deliverability of , and to diagnose failures that have occurred. Thankfully, the new DMARC standard allows mail senders such as Xero to receive delivery reports from the recipient mail servers such as GMail, Yahoo! or Hotmail, and actually diagnose any issues that might have occurred. We are very excited to have partnered with Agari to begin the process of moving all our invoicing to conform to the DMARC standard. Agari were a part of the original DMARC working group, and have worked with some enormous US brands to share their advice and implement cutting edge trust solutions. The DMARC standard allows us to conclusively prove to recipient mail servers the invoice s we send have come from a legitimate source. It blends the older SPF and DKIM standards with a new alignment process to make sure that their protection messages apply to all aspects of an , not just the Envelope Sender address. The other thing that DMARC and Agari allow us to do is to track trends in our delivered , and to work on errors in our configuration as reported by the recipient. We re now at the point where Google, Hotmail and other mail providers can send us daily reports telling us why they rejected , and of any issues that they ve had delivering to their users. This is unprecedented and a completely new approach to give us visibility into what happens to the we send once it leaves our server. Our Roadmap: Changing the way we send is difficult, but also risky. We absolutely want to avoid making our customers invoice s fail to deliver. To this end, we re gradually phasing in our adoption of DMARC and the changes to our sending process. Step 1: Monitor delivery using Agari tools

8 May 2013 We have implemented a DMARC policy in report only mode, which allows us to track the changes we re making without instructing recipient mail servers to reject any that doesn t conform. Step 2: Change the from address of outgoing invoice s to address. June 2013 This lets us use a unique domain name for invoice s that isn t reliant on the reputation or delivery characteristics of any other that Xero sends. It also allows us to introduce simpler SPF and DKIM policies, and DMARC policies, because any changes will be restricted just to that comes from a post.xero.com address. This also means that the visible From address now aligns with the Envelope Sender address, as they are both xero.com sub domains. (Domain alignment is a DMARC requirement) Next steps: Step 3: Implement DKIM signing of post.xero.com This will allow us to cryptographically sign all invoice s, so that a recipient mail server knows that they definitely came from a Xero server and aren t spam. Step 4: Turn on DMARC enforcement This will allow us to instruct recipient mail servers to start enforcing our DMARC policies, meaning that they will definitively know whether we ve sent is from Xero, from a mail server we control, and hasn t been tampered with. This will allow mail providers such as Gmail or Yahoo! to have a much higher level of confidence in our , and will short circuit a lot of their spam filtering. The end result of these four steps will be that we ll have a much higher quality of that we re sending, recipient mail servers will have a much higher confidence in the they receive, and we ll have industry leading monitoring of the deliverability of the we send, so we can be confident that Xero s customers are sending invoices to their customers successfully! Frequently Asked Questions: Is it possible to have s come from our mail domain rather than post.xero.com?

9 Unfortunately, there is no longer a way to do this and reliably deliver to your customers. But it looks like Xero is sending the to my customer, and it messes up their plugins and ruins our brand! We understand, and sympathise with this concern but currently don t have a technical solution that will allow us to do this. Is it possible to have all our outgoing s appear in our Gmail Sent Items folder? To do this, we either have to send out your invoice s via your Gmail account (which we re not able to do), or to carbon copy (CC) your account with a copy of every message you send from Xero. We could consider making the CC of your account on every Xero generated an option that you can configure, if that s something that the community votes on as a good feature. In the meantime you can click the Send me a copy checkbox when sending invoices. Another option we have is to track all outgoing messages in the Xero Notifications Inbox, so that you can view all messages sent from your account there. Rapportive and other mail client plugins don t display our company information any more This is because these tools are looking at the displayed From address to determine which company has sent the . Unfortunately there is no way we can change the way we deliver to allow these plugins to treat originating from the Xero mail servers as coming from you. Is it possible to change the address from message service@post.xero.com to noreply@post.xero.com to make it more obvious it s unmonitored? That s an interesting question, and one we considered when setting up the From address. We may consider changing this in the future. Does Xero track the deliverability of our ? Yes. We are in the early stages of using the Agari platform to provide us with metrics and dashboards showing us the deliverability of the invoice s you send using Xero. Please note that this is not a privacy invading tracking system like the hidden image / web bug approach used by some delivery tracking systems, and is purely a report delivered from the recipient mail server (e.g. GMail) to our secure Agari environment.

10 Does Xero return bounce messages to my address? As mentioned in the Bounce section above, there s two different types of bounce messages that we may receive in response to your invoice s, as well as other replies that might come to our servers rather than directly to you. For all of these message types we ll attempt to redirect the bounce or reply back to your inbox. In those situations where that s not possible the other address will receive the Oops! message above. However note that there are many different ways for to fail silently and, in some cases, neither us nor you will be notified by the recipient mail server. Why haven t you set up SPF for post.xero.com? Is the SPF record for xero.com valid? The domain used for SPF checks of our invoice s sent through Xero is notifications.xero.com. The SPF record for this domain is relatively simple, and we have had it s configuration checked by external parties. Are more messages being treated as spam than before these changes? With any change to the way we send , we re very cautious and want to make sure that we re making things better rather than worse. Unfortunately, before embarking on this project with Agari and the DMARC standard, we weren t able to scientifically track deliverability of like we are now able to. We were however able to unscientifically gather metrics about delivery issues through support tickets raised by our customers, and we re confident that these have decreased since the changes. Going forward, we look forward to being able to collect better metrics on delivery, and depending on the community demand we may even be able to share some of these metrics with Xero customers. Could we give each Xero customer their own sub domain (e.g. my company.xero.com)? That s an interesting idea, and not one that we ve yet explored in depth. In that case, we d need to provide signing, DKIM, SPF and redirection for hundreds of thousands of custom domain names, potentially one per customer. We re unlikely to do this in the short term. I previously used another desktop product, and it was able to send on my behalf. Why can t Xero do this? Most previous generation or desktop products that sent would typically send the directly from the logged in user s computer, using their local Outlook or Mail preferences. Because Xero is a Software as a Service application running on our servers, this model is not

11 possible our servers need to be able to send your invoice even if your computer is not connected to the internet. You could potentially revert back to sending your invoice s from your desktop by downloading the PDF files and sending them from your local Outlook or Mail client, or send the invoice s to yourself from Xero and forward the invoice s to your customer manually from your mail client. However, this process will be more time consuming. I use another SaaS product, and it is able to send on my behalf. Why can t Xero? Our review of other SaaS applications, and the ways they send on behalf of the users (impersonate), shows two main approaches: 1. Incorrect ing practices, where impersonated will fall foul of the SPF or spam rules we ve discussed above, much as our solutions did. 2. Correct ing practices, but which require the customer to set up SPF and DKIM records in their DNS, and don t allow impersonation until the SaaS provider has confirmed these settings are correct. We now clearly hear that some of our customers would like the second option, and to be able to opt in to the extra configuration work to configure their SPF and DKIM to allow us to send on their behalf. We are currently evaluating the provision of this feature, and the engineering and support effort required to provide it. We have run some analysis on the number of customers we expect would be able to do the required configuration to make this work correctly, but it would be still great to hear your feedback via our Xero Community site.