How To Send. Zillions A DAY. (And Keep Customers Happy and Engaged) by Mike Hillyer, Senior Director of Global Solution Consulting

Transcription

1 How To Send Zillions OF S A DAY (And Keep Customers Happy and Engaged) by Mike Hillyer, Senior Director of Global Solution Consulting

2 How Much Does Your Company Send? A hundred a day? A thousand a day? More? Believe it or not, many organizations send several million s a day, and enterprises, large online retailers, social networks and gaming sites often send as much as a billion a month. In the case of social networks, a few send as many as a billion s a day. No matter how much companies send, they all send for the same reason: They have a community of customers or users, and these companies need to keep their communities engaged and in-the-know. Why? Because a disengaged community is no community at all, and a disengaged customer base is not going to be your customer base for long. Is it easy to send high volumes of ? Yes and no. It s easy to do, but the way you do it is what matters. Large volumes of mail can be expensive to deploy. Worse, large volumes of mail may be perceived as spam by ISPs. If done incorrectly, high-volume sending can drain your resources and your revenue, and provide no value at all to your customers. How can you keep your customers happy and engaged if your message never makes it to the intended inbox? Here are four steps you need to take to do it right: Consider Your Hosting Options As a sender, you have the option of using an service provider (ESP) or using your own sending infrastructure, and you have Marketing Reports is a great source for reputation building. You ll find a wealth of information, original articles, reviews and links to help you start or improve your opt-in marketing. -marketing-reports.com to choose the best model for your situation. ESPs provide not only infrastructure but also expertise, to handle the myriad details of sending for their clients. This enables companies to focus on their core business rather than the many technical challenges of high-volume . Nonetheless, you should look carefully at the cost/benefit trade-off of in-house versus outsourced sending, especially as your volume begins to grow. Sending in high volumes can quickly become costly when working with ESPs that charge on a CPM (Cost Per Mille [thousand]) basis. If you choose to operate your own sending infrastructure, there are benefits to directly controlling your own database. You will typically be in a position to better understand your customer s behavior across business units, making it far simpler to respond to customers preferences and message them in their preferred format. Just be sure to put a database administrator on your team so you can actively fine-tune your sending strategies over time for optimal results. Operating your own sending infrastructure can offer other benefits, such as sending mail with lower latency and more straightforward synchronization with your database of record. In the end, it s not a matter of insourcing versus outsourcing but of right sourcing, or doing what makes the most sense for you and your customers. You may very well find that a mixed environment works best, in which both you and an ESP share the sending responsibility. Improve Your Sending Reputation The higher your sending volume, the more you need to improve your sending reputation among the ISPs that handle most of the incoming traffic on the Internet. This holds true for major ISPs such as MSN, AOL, Yahoo, AT&T, Gmail, EarthLink, Cox, Bell South etc., as well as smaller local ISPs. A high reputation is critical in getting the majority of your mail safely in the hands of your intended recipients. To build and maintain a good reputation you need to follow best practices for list acquisition, list hygiene, segmentation and relevancy. This guide focuses on the technical requirements for sending high volumes, but keep in mind that doing so without a solid reputation on your domain and sending IPs is very difficult. From a technical perspective there s a number of bases we need to cover regarding authentication, whitelisting, bounce processing and complaint handling. Authentication. As a reputable sender you will want to associate your IP addresses with your domain using the authentication standards available to you. These include SPF, SenderID, DomainKeys (DK) and DomainKeys Identified Mail (DKIM). There are indications that SPF (and SenderID by association) is ineffective, but given the low effort required to implement it, we recommend doing so. While SPF and SenderID are purely DNS-based, DK and DKIM require an implementation either during message creation or during relay by the MTA and as a result will impact the maximum throughput of your infrastructure. DK is quickly being superseded by DKIM, but with most solutions supporting them it is simple enough to use both when sending to an ISP that supports one or the other. Implementation details will vary based on your sending solution. While some recommend selectively signing DK and DKIM only for messages sent to ISPs that are known to check authentication (in order to lower the impact signing has on throughput on a solution that takes a significant performance hit from signing), it is recommended that you sign all messages; you never know who is checking for authentication without announcing it. Whitelisting. One benefit of getting on the various whitelists provided by ISPs and reputation providers is that in some cases you can send higher volumes on whitelisted IP addresses than would otherwise be possible. Keep in mind that in most situations whitelisting comes after sending has already begun in order to allow the provider of the whitelist to examine your sending patterns as part of the whitelisting process, so put your best foot forward (and follow it up with consistent behavior). Bounce Processing. One quick way to lose reputation is to repeatedly send mail to invalid addresses. ISPs will track how many non-existent addresses you send to and throttle you accordingly. Even more seriously, ISPs will occasionally take inactive addresses and re-activate them as spam traps; any mail sent to the address will immediately get classified as a spam source and will be filtered accordingly. To prevent this from happening it is necessary to capture and act on the responses sent by the ISPs and unsubscribe those addresses identified as non-existent or inactive, while retaining those with responses that identify users on vacation, with full mailboxes, and other non-fatal conditions. Commercial sending solutions will perform this step automatically with varying levels of effectiveness, while other platforms will require a third-party solution. Complaint Handling. In an effort to help senders improve their practices, a number of ISPs have implemented feedback loop (FBL) programs using the abuse reporting format (ARF), the industry standard format for spam reporting. When a user on a supported ISP clicks the This Is Spam button, an automated message is sent to an address defined in advance (when signing up with the ISP for the feedback loop program). By processing these messages and unsubscribing the relevant users, you prevent further reputation 2 3

3 A high reputation is critical in getting the majority of your mail safely in the hands of your intended recipients. damage that may result when sending them future messages. You can also use this feedback to refine your campaign objectives and approach, to provide your customers with the content and offers they are most likely to need or want. The ARF format used by the ISPs makes it relatively straightforward to process feedback loop messages and use them to unsubscribe the users who have complained about your messages. Some sending solutions can handle FBL messages natively while others require separate tools that can open ARF-formatted messages. Improve Your Sending Infrastructure Several key architectural components come into play to make it possible to send high volumes of , including network connectivity, server hardware and software. Network Connectivity. Most professional sending operations are based in rented Figure 1: Server Hardware INTERNET datacenters, simplifying the provisioning of network connectivity. For example, if we need to send a billion s per month, we re talking 2.5 million messages per hour, and we ll assume a sample message size of 50 kilobytes (51,200 bytes), meaning that we need to send at a rate of 2,500,000 * 51,200 = 128,000,000,000 bytes per hour or megabits per second. With this throughput, you will certainly need gigabitspeed networking in the datacenter. Do the math on your own expected throughput, to determine your datacenter needs. But keep in mind that has a significant impact on your bottom line so you won t be able to tolerate extended outages. You need to not only make sure that the datacenter you choose has redundant power and backbone connections, you also need to consider using redundant datacenters. Server Hardware. Moving high volumes of does not require the purchase of custom server hardware but it does require making a proper investment in hardware. Generally speaking you will be using an infrastructure similar to the one shown in Figure 1. The Message Injector queries the database and uses the results to assemble one or more messages, which it relays to the Outbound Mail Server. The Outbound Mail Server queues the message, performs any necessary manipulations on the message and then sends it via the Internet to the Inbound Mail Server Outbound Mail Server Database Server Message Injector destination server. In the event of a delivery failure message or a feedback loop message, the incoming message arrives via the Internet to the Inbound Mail Server. The Inbound Mail Server performs antivirus/ anti-spam scanning and then, in the case of a legitimate message, processes the message and updates the subscriber information in the database. Note that not all server solutions can perform this processing in-stream. When using such solutions, an intermediate server will be needed to accept the clean message from the Inbound Mail Server and process it using custom code. In a production deployment there can be several variations on this example, typically with multiple servers used on the outbound and inbound roles, with multiple message injectors pushing to the outbound machines and often specialty servers on the inbound side dedicated to processing incoming feedback loop and bounce messages. To send in very high volumes (say, more than a billion a month), quality working mail servers should feature the following: 2x multi-core, 64-bit processors 16-32GB of RAM 8x 15K RPM hard disks Battery-backed RAID-10 controller The specific details of your hardware selection will depend on your actual and expected sending volume and the ability of your specific software to leverage the resources provided. A large number of fast disks in a RAID-10 array is recommended for the message spool as standards-compliant mail servers must write messages to disk before accepting them for delivery, placing significant demands on storage resources. Software. For many years, the first choice for sending software was open source message transfer agents (MTAs), the most widely used of which were Postfix and Sendmail. The problem with open source MTAs is that their performance tops out at around 100,000 messages per hour. To support a one-million-per-hour rate, for example, you d need 10 individual servers, each running an instance of an open source MTA. Commercial sending solutions typically show real-world performance ranging from 500,000 messages per hour per server to over two million messages per hour per server. Check your reputation. Return Path, a leading certification and scoring company, offers senders, receivers or consumers free access to Sender Score, an online reputation checking tool. senderscore.org In addition, your sending software should support IP domain segregation. ISPs pay close attention to the source IP domains of flowing into their servers. If any domain sends a high rate of mail that looks like spam, or gets flagged as spam by recipients, or contains bad addresses, the ISP will filter traffic from that domain accordingly, or might block all traffic from that domain outright. If all of your mail comes from a single IP, you run the risk of all of your mail getting blocked. For example, let s say we have one million messages to send today, and they break down into three groups. The first third is going to addresses that we re very confident are legitimate because those addresses have been used in the past to send invoices, order confirmations and other transactionbased s. The second third of the list was generated with confirmed- or doubleopt-in tactics, so we re confident that these 4 5

4 won t be blocked, but not with the same level of certainty as group one. The final third of the address list was built using unconfirmed- or single-opt-in tactics; we need to separate out these addresses because they re likely to generate a higher number of spam complaints than group one or group two. Given our situation, we would want sending software that could accommodate at least three IP domains, but keep in mind that companies that send very high volumes of mail often need many more. For instance, large service providers that manage campaigns on behalf of numerous brands would, ideally, want the ability to create unlimited IP addresses. A number of tools for checking the reputation of your IP addresses can be found at Word to the Wise, a software provider that helps ISPs and ESPs to manage spam complaints and deliverability issues. wordtothewise.com Some sending software goes beyond basic message queuing and sending, providing the additional functionality required for a high-volume sender. Here are some other critical features to look for: High Availability. With high-volume sending, you absolutely need a solution that provides high availability out of the box. If a server goes down you can t afford to be frantically activating a warm spare, just to find out that it too has some issue. You need an active-active solution that reacts automatically to server failures and keeps the mail flowing. Centralized Management. You need a solution that can be easily managed on your terms, whether you prefer editing configuration files or using a web interface. In addition, you need something that grows with you, providing centralized management of an entire cluster of servers. Advanced Reporting. One key to successful sending at high volumes is keeping tabs on how your server is performing and how your mailings are doing. You need to know what is passing through your server, how quickly messages are moving, whether queues are backed up, how the various ISPs are treating your traffic, all with the ability to drill down on specific source IPs and specific destination ISPs. You need to be able to see all of this in real time and across your entire infrastructure. When choosing sending software, be careful not to choose a free or low-cost solution only to spend countless hours building workarounds, or writing scripts to automate administration or provide redundancy. This work requires time, and time spent creating tools that are already provided in an alternative solution is time (and money) wasted. You are always better off using your time to create your special sauce, that which makes your business unique and gives you a competitive advantage. Strengthen Your Monitoring Capabilities When sending high volumes of mail, you just can t afford to have a fire and forget mentality. You need to keep a constant eye on your environment, monitoring multiple key factors to ensure that you can continue to successfully send. Reputation Monitoring. Remember earlier when we said you needed to improve your sending reputation? You also need to maintain your good reputation. To do that you should take advantage of reputation monitoring tools provided by companies like Return Path ( and you should also keep a close eye on the reports produced by your sending software. You need to watch metrics such as bounce rates, FBL hit rates, blacklist hits, transient failures and response rates. Infrastructure Monitoring. You need to make sure your keeps flowing no matter what happens, and that means making sure your infrastructure stays online. You will need to monitor the health of the servers that support your infrastructure, the network components that carry your messages and the software that creates and relays your messages. There are a number of monitoring solutions available to accommodate any platform and budget. Make sure to test simulated failures to confirm that monitoring is working successfully. Consider setting up a simulated mailing that runs on a regular basis using your full infrastructure stack: a monitoring script can check an inbox and if the test message fails to appear, something is potentially wrong in your sending infrastructure. This approach can help identify issues that may pass by other monitoring systems unnoticed (and can be integrated into some monitoring solutions directly). Response Monitoring. Keep in mind that you get what you monitor for; if you focus too much on one metric it may improve without helping the big picture. In addition to making sure all the underlying pieces are in place, don t forget to keep an eye on things where the rubber meets the road. You may be sending at phenomenal rates with great metrics but failing to generate customer actions that lead to revenue. So don t forget to monitor your response rates. Start Sending! While by no means an exhaustive list, this should give you a good idea of what s required in a high-volume sending environment, while keeping your customers happy and engaged. If your company is growing, it pays to see the road ahead, and high- volume sending might be in your future sooner than you think. About Message Systems The industry leader in messaging technology, Message Systems offers a family of software solutions and services that addresses the digital communications needs of today s most innovative companies. Telecommunications carriers, ISPs, marketing services providers, cloud computing firms and social media companies rely on Message Systems software to power the message-based communications driving their critical business initiatives. Message Systems solutions get billions of unique messages to the right place at the right time every day through the full range of channels: , SMS text, MMS messaging and more. Founded in 1997, the company is headquartered in Columbia, Maryland. About the Author Based in Calgary, Canada, Mike Hillyer has managed the Enterprise Sales Engineering team at Message Systems since He also serves as a committee co-chair for the Online Trust Alliance and is an active author and speaker, providing insights and information on infrastructure and deliverability. You can follow him on 6 7