Bulk Verification Documentation

Transcription

1 Bulk Verification Documentation Release verifyapi.com January 14, 2015

2

3 Contents 1 Product Overview Product Overview QuickStart Quick Start Features Features FAQs Frequently Asked Questions Status Codes Verification Response Codes For Bulk Verification Glossary Glossary i

4 ii

5 Our validation service will help your business reduce costs by eliminating bad addresses from your information systems. All you need to do is upload your list and wait for the results to come in. Contents 1

6 2 Contents

7 CHAPTER 1 Product Overview 1.1 Product Overview Why Address Verification? Sending s to non-existent addresses is never a good idea for many reasons including: Damaging the reputation of your ESP infrastructure (IP addresses) Many ESPs will have maximum hard bounce thresholds. Exceed these and your account could be closed. Bad s in a list negatively affects deliver-ability rates A Block List can keep track of failed deliveries (e.g. bad address) at a particular domain. Too many failed deliveries can raise some flags thus raising the risk of your campaign being flagged as spam. Processing hard bounces means wasted time An sent to a bad or non-existent address results in a bounce message coming back. Someone or something (e.g. automated software) is required to remove the bounces from a list so as to avoid sending to bad addresses next time. Tip: Cleansing bad s from a list using verification technology before embarking on an ing campaign significantly reduces the above challenges Product Description verifyapi.com offers data cleansing services specifically for addresses. Offering a fully cloud based solution, verifyapi.com provides fast, reliable and accurate verification. With unrivalled coverage in all areas (including traditionally hard to verify addresses such as Hotmail and Yahoo), verifyapi.com is the natural choice for customers requiring good coverage in their applications for both B2B and B2C addresses Reliable and Secure Security of your data is our primary concern. To ensure security, we implement: Robust data security implemented using Microsoft Trustworthy Computing guidelines. Logical security of your data by allocation of your own, dedicated database. Any data that you upload goes into your database and is not visible by anyone else. Physical security in secure US and UK data centers. 3

8 Once you delete a list from our system, it is wiped completely off our servers. Ultimately, no one gets access to your data except you. As well as the assurance of security, we appreciate that you want reliability. That s why, as a customer, you receive priority support by phone and . In the unlikely event of a technical problem, our promise to you is to resolve your issue within 24 hours. Our systems include built in exception and error alerting. This means that, in the unlikely event that you need to call us, we ll already know about the problem and be well on the way to fixing it How To Get Started To get started create a new account and upload your list. 4 Chapter 1. Product Overview

9 CHAPTER 2 QuickStart 2.1 Quick Start This quick start guide is designed to get you up and running as fast as possible. Please follow the steps below in sequence: ) Create an Account Create an account ) Upload File Navigate to the Upload a file tab. You can drag and drop or manually upload any.csv (Comman-Seperated Values) or.txt (Text File) file. This file can contain multiple data fields, not just addresses. Our system will pick out the address field automatically and preserve the rest of your data ) Choose your Payment Method You can either pay as you go or subscribe to the service for a set number of verifications per day ) Start Verification Please not that bulk lists can take up to 24 hours to verify ) View your Results! 5

10 6 Chapter 2. QuickStart

11 CHAPTER 3 Features 3.1 Features Fanatical Service Quality Management (SQM) verifyapi.com operational staff obsessively monitor services to ensure the best possible uptime and coverage. Uptime and functional correctness is actively monitored on a minute by minute basis from multiple data centers dispersed across North America and Europe Multi Factor Verification Progressive verification using multiple verification processes including: Syntax checking DNS checking Mailbox checking Unrivalled Coverage With more than 5 years experience and the benefit of owning our own software stack, verifyapi.com has refined its services over the years to provide good coverage not only of the easier B2B domains but also the more technically tricky B2C domains including: Hotmail Yahoo AOL Yandex Spam Trap Detection After many years R&D, verifyapi.com has developed various secret sauces that can effectively detect and eliminate spam traps from several well known Block List. 7

12 3.1.5 Disposable Address Detection Detect and eliminate DEA What it does verifyapi.com is used to check addresses in bulk. Not only are syntax and domain checked, but that the user mailbox is available too. This is the only way to know for sure if an address is valid. Additionally identified as part of the verification process is extra information including: DEA. Spam Trap How it works addresses are verified using various filters and processes. As a high level overview, an address submitted for verification goes thorough the following filters: Syntax A basic inspection of the systax of the address to see if it looks valid. Work is done only using server CPU (Central Processing Unit) based on simple pattern matching algorithms. DNS A Verifies a domain exists in DNS. Domains that do not exist in DNS cannot have mail servers or boxes. DNS checks are performed over the network. DNS MX Verify MX records using DNS. Domains that do not have MX records, have no mail servers and therefore no valid boxes. MX checks are performed over the network. MailBox Verify boxes with SMTP checks. Connect to mail server and perform SMTP protocol to verify if mail box exists. This is the deepest level of verification. It is performed over the network. 8 Chapter 3. Features

13 CHAPTER 4 FAQs 4.1 Frequently Asked Questions How fast is it? Please allow up to 24 hours for a list to process How do I get s into the system? Upload any text (.txt) or CSV (.csv) file that contains addresses. This includes Excel (export as.csv), Outlook (export as.csv) or any other application that can support exporting as text (almost all applications support this). Your files can contain extra data fields, we simply append our results to the end of your existing file How do I get results out of the system? You can export results in the same format that you imported them Are there any s that can t be checked? For servers that we know give problems, such as those that give false positives (catch-all), we flag the address appropriately How often should I check addresses? For large lists (i.e. > 10,000), we suggest checking every 2 weeks. Smaller lists can be checked monthly addresses are always changing. People change addresses as they (for example): Move Internet providers Move to a new employer Change name (e.g. after getting married) If your business relies on lists for its marketing, it s important that you keep the lists current and clean to maximize your target potential and reduce the overheads associated with managing too many bounced s. This is why we recommend frequent checking. 9

14 4.1.6 How do I sign up? To get started create a new account and upload your list How secure is my data? Very. We take your security and confidentiality seriously. We know that your lists are confidential and we do everything we can to make sure that they stay that way! Your data is held securely in US and UK data centers. As well as physical security of locked server rooms, we use strong password security in our software services. This means that no-one other than you can see your data. Also, if you choose to cancel your plan at any time, we undertake to destroy / delete your data so that it cannot be retrieved or viewed by any other parties Do you share my lists or use them in any other way? No. No-one other than you has access to your data / lists What browsers do you support? Generally, anything that is younger (as of December 2014) than 7 years old. Including: Safari Internet Explorer (version 7+) Firefox (version 2+) Chrome Opera As this is an Internet based service, it means that it runs happily on any operating system including Mac, PC, Linux, Unix etc Can it do Hotmail? Yes Can it do Yahoo? Yes. Based on our own internal testing verifyapi.com is currently the only verification service to offer effective and repeatable coverage for Yahoo addresses Can it do Yandex? Yes it can. 10 Chapter 4. FAQs

15 Can it find spam traps? Partially. A Spam Trap is a moving target. In theory (and indeed in practice) anyone can setup a Block List and start putting spam traps into the wild. verifyapi.com has Spam Trap detection capabilities that covers several of the well known block lists. Whilst it is not possible to deliver 100% coverage of all spam traps from all block lists, verifyapi.com provides the best Spam Trap detection capabilities available How does it work? At a basic conceptual level, the process of verifying addresses is very simple. Google for Send using telnet for a quick and general overview of how it s done. To verify an address without sending an , simply go as far as the RCPT TO stage and parse the response code. That s the easy bit and can be accomplished in just a couple of dozen lines of a PHP script! The hard bit is dealing with mail services that are intrinsically configured to work against the process of verification or any similar SMTP based activity. The reason that any / SMTP process is difficult from a client perspective is that mail services need to protect themselves from an ever increasing landscape of abuse including spam and DDoS attacks. verifyapi.com s strength in dealing with the hard bit of verification comes from years of experience in doing verification together with our complete ownership of our SMTP verification software stack together with an extensive cloud based infrastructure. That s why verifyapi.com can do the hard bits best and offer outstanding coverage on the more difficult domains such as Yahoo and Hotmail Can I get blacklisted using this service? No. It s verifyapi.com infrastructure that does the work Will anyone know that I am verifying their address? No. It s verifyapi.com infrastructure that does the work Your service says an address is OK and I know it s Bad (or vice versa)? verifyapi.com queries mail servers in real time. Mail servers respond with one of 2 possible answers for a given address: Yes, the address exists - SMTP code 2xx No, the address doe not exist - SMTP code 5xx verifyapi.com uses the above response codes to determine if an address is valid or not and reports this back to you. This method of determining address validity works in >99% cases. However, nothing is guaranteed. In a small number of cases it is possible for a mail server to report one thing on verification and do something different on trying to deliver an to the address verified. For example, say we take an address of [email protected] to send to. We are sending from a fictitious address [email protected] Frequently Asked Questions 11

16 reports with status code of OK from the verification service. However, when you send an to the bounces. Further inspection of the bounced NDR (Non Delivery Report) headers show something like the following: Delivered-To: Received: by with SMTP id n6csp24867ioo; Sat, 6 Jun :57: (PST) X-Received: by with SMTP id 5mr oie ; Sat, 06 Jun :57: (PST) Return-Path: <> Received: from SNT004-OMC2S34.hotmail.com (snt004-omc2s34.hotmail.com. [ ]) by mx.google.com with ESMTPS id ws5si obb for <[email protected]> (version=tlsv1.2 cipher=ecdhe-rsa-aes128-sha bits=128/128); Fri, 6 Jun :57: (PST) Received-SPF: none (google.com: SNT004-OMC2S34.hotmail.com does not designate permitted sender hosts) Authentication-Results: mx.google.com; spf=none (google.com: SNT004-OMC2S34.hotmail.com does not designate permitted sender hosts Received: from SNT004-MC2F40.hotmail.com ([ ]) by SNT004-OMC2S34.hotmail.com over TLS secur Fri, 6 Jun :57: From: [email protected] To: [email protected] Date: Fri, 6 Jun :57: MIME-Version: 1.0 Content-Type: multipart/report; report-type=delivery-status; boundary="9b095b5adsn=_01d010aabce2c5cc0008c930snt004?mc2f40.ho" X-DSNContext: 335a7efd Message-ID: <mjz7zgtpi @snt004-mc2f40.hotmail.com> Subject: Delivery Status Notification (Failure) Return-Path: <> X-OriginalArrivalTime: 06 Jun :57: (UTC) FILETIME=[CEAD2EE0:01D0114B] This is a MIME-formatted message. Portions of this message may be unreadable without a MIME-capable mail program. --9B095B5ADSN=_01D010AABCE2C5CC0008C930SNT004?MC2F40.ho Content-Type: text/plain; charset=unicode-1-1-utf-7 This is an automatically generated Delivery Status Notification. Delivery to the following recipients failed. [email protected] The header of the NDR shows that Hotmail thinks the address is invalid as far as sending to this address is concerned. However, Hotmail reports that the same address is valid as far as the verification activity performed by verifyapi.com. The discrepancy in verification results versus mail send is with the Hotmail infrastructure reporting one thing but doing the exact opposite. This behaviour occasionally (particularly from Hotmail) is seen in a small amount of cases and is attributable to internal Hotmail (or other mail services) system anomalies. The majority (>99%) of verification status versus mail send is consistent. However there are some edge cases caused by system faults in the mail service providers themselves. For these small number of cases, there is nothing that can be done at the verification stage. 12 Chapter 4. FAQs

17 How do I cancel a subscription? For pay as you go accounts, the account is free and you just pay when you need it. For subscription plans, you simply need to contact us via the support button in your account dashboard, and your cancellation will take effect at the end of the month. Please note that subscription plans have a minimum 2 month period, after that you can cancel at any time. Our service is so good that we hope that you won t need to cancel! However, if you do, we are proud of our simple cancellation process. Unlike many other companies, there are no hassles with fine print and tricky notice periods. Once you ve cancelled your service, we ll disable your account and make sure that all your data is securely disposed of (deleted) I have received x percentage of bounce backs using s you verified If you experienced a large number of NDRS (Non Delivery Reports) using addresses that were verified OK by our service there may have been an issue with your Service Provider (ESP) or the target mail server. Due to the nature of our verification process a mail box reported as OK can definitely accept incoming s. This is due to a target mail server existing, the server not being a catch all and the mail box accepting incoming mail. If you experience a bounce back with your validated OK addresses, there has likely been a fault with your ESP or on the target mail server. We are happy to work with customers and their ESPs in order to resolve such issues Frequently Asked Questions 13

18 14 Chapter 4. FAQs

19 CHAPTER 5 Status Codes 5.1 Verification Response Codes For Bulk Verification Main Status Codes Ok Verification passes all checks including Syntax, DNS, MX, Mailbox, Deep Server Configuration, Grey Listing Bad Verification fails checks for definitive reasons (e.g. mail box does not exist) Unknown Conclusive verification result cannot be achieved due to mail server configuration or anti-spam measures. See table Additional Status Codes Additional Status Codes None No additional information is available. This status differs from other additional status codes as it should not be retried (the result will not change). AtSignNotFound The sign is not found in address. CatchAllConnectionFailure The secondary catch all verification failed upon connection. CatchAllValidationTimeout The secondary catch all validation timed out. DnsConnectionFailure The DNS connection failed. DnsQueryTimout The DNS service failed to respond (timeout). A typical example of this status code is when a DNS server on the target infrastructure is configured incorrectly. DomainIsInexistent The domain (i.e. the bit after character) defined in the address does not exist, according to DNS records. A domain that does not exist cannot have boxes. A domain that does not exist cannot have boxes. DomainIsWellKnownDea The domain is a well known Disposable Address DEA. There are many services available that permit users to use a one-time only address. Typically, these addresses are used by individuals wishing to gain access to content or services requiring registration of addresses but same individuals not wishing to divulge their true identities (e.g. permanent addresses). 15

20 DEA addresses should not be regarded as valid for send purposes as it is unlikely that messages sent to DEA (Disposable Address) addresses will ever be read. DEA addresses should not be regarded as valid for send purposes as it is unlikely that messages sent to DEA addresses will ever be read. DomainPartCompliancyFailure The domain part compliancy failed. DoubleDotSequence The addresses domain contained double dots. IpBlockDetected An IP block has been detected. InvalidAddressLength The address is an invalid length. InvalidCharacterInSequence The address contains an invalid character. InvalidFoldingWhiteSpaceSequence The address contains a folding white space. InvalidLocalPartLength The address contains an invalid local part length. InvalidWordBoundaryStart LocalSenderAddressRejected The local sender address was rejected. MailboxFull The mailbox is full. Mailboxes that are full are unable to receive any further messages until such time as the user empties the mail box or the system administrator grants extra storage quota. Most full mailboxes usually indicate accounts that have been abandoned by users and will therefore never be looked at again. We do not recommend sending s to addresses identified as full. MailboxDoesNotExist The mailbox does not exist. 100% confidence that the mail box does not exist. MailboxTemporarilyUnavaible The mail server is operating Grey Listing. Whilst we endeavour to verify grey listed addresses, sometimes it is not possible whilst still offering timely verification results. NoMxServersFound There are no mail servers defined for this domain, according to DNS. addresses cannot be valid if there are no servers defined in DNS for the domain. OperationCanceled The operation was cancelled. ProxyConnectionTimeout The proxy timed out. ServerDoesNotSupportInternationalMailboxes The server does not support international mailboxes. International boxes are those that use international character sets such as Chinese / Kanji etc. International boxes require systems in place for Punycode translation. Where these systems are not in place, verification or delivery is not possible. For further information see Punycode. ServerIsCatchAll The server is configured for catch all and responds to all verifications with a status of Ok. Mail servers can be configured with a policy known as Catch All. Catch all redirects any address sent to a particular domain to a central box for manual inspection. Catch all configured servers cannot respond to requests for address verification. SmtpConnectionFailure The TCP connection to the target mail server failed. SmtpConnectionShutdown The target mail server prematurely terminated the connection. 16 Chapter 5. Status Codes

21 SmtpConnectionTimeout A timeout occurred whilst waiting for a connection to the target mail server. SmtpConnectionRefused The SMTP connection was refused by the remote server. This status code applies to all Office 365 mail server, stopping verification, as these mail servers all operate a catchall policy. Success Successful verification. 100% confidence that the mail box exists. TcpSocketUnavailable The TCP socket is unavailable for date exchange. TooManyAtSignsFound Too signs found in address. Only character is allowed in addresses. UnableToBindToLocalIpAddress Cannot bind local IP endpoint specified. UnexpectedQuotedPairSequence UnhandledException Transient service fault. Unknown The reason for the verification result is unknown. PossibleSpamTrapDetected A possible spam trap address or domain has been detected. Spam traps are addresses or domains deliberately placed on-line in order to capture and flag potential spam based operations. Our advanced detection heuristics are capable of detecting likely spam trap addresses or domains known to be associated with spam trap techniques. We do not recommend sending s to addresses identified as associated with known spam trap behaviour. Sending s to known spam traps or domains will result in your ESP being subjected to blocks from a DNS Block List.. An ESP cannot tolerate entries in a Block List (as it adversely affects deliver-ability for all customers) and will actively refuse to send s on behalf of customers with a history of generating entries in a Block List Verification Response Codes For Bulk Verification 17

22 18 Chapter 5. Status Codes

23 CHAPTER 6 Glossary 6.1 Glossary B2B Business To(2) Business Business hosting services are generally private, enterprise grade hosting services typically hosted in either private data centers or in cloud based infrastructure such. Business to business refers to the activity of businesses sending to clients using business addresses. B2C Business To(2) Consumer Consumer hosting providers are generally well known, mostly web based providers such as Hotmail, Yahoo, AOL, Gmail etc. Business to consumer refers to the activity of businesses sending to clients using consumer addresses. Verifying addresses in consumer domains is generally more technically challenging than B2B Block List See DNSBL. DDoS Distributed Denial of Service See Wikipedia - Denial-of-service attack for more information. DEA Disposable Address There are many services available that permit users to use a one-time only address. Typically, these addresses are used by individuals wishing to gain access to content or services requiring registration of addresses but same individuals not wishing to divulge their true identities (e.g. permanent addresses). DEA addresses should not be regarded as valid for send purposes as it is unlikely that messages sent to DEA addresses will ever be read. DEA addresses should not be regarded as valid for send purposes as it is unlikely that messages sent to DEA addresses will ever be read. DNS Domain Name System At its simplest level, DNS converts text based queries (e.g. a domain name) into IP addresses. DNS is also responsible for providing the MX records needed to locate a domains mail servers. See Wikipedia - Domain Name System for more information. DNSBL DNS Block List 19

24 As an anti-spam measure, mail servers can use spam black lists to look up the reputation of IP addresses and domains sending . If an IP or domain is on a block list, the mail server may reject the senders message. See Wikipedia - DNSBL for more information. ESP Service Provider A service that sends s on your behalf. See Wikipedia - service provider (marketing) for more information. Grey Listing A technique used in mail servers as an anti-spam technique. Sometimes also known as deferred, grey listing arbitrarily delays the delivery of s with a try again later response to the client sending the . MX See Wikipedia - Grey Listing for more information. Mail Exchanger The MX is a server responsible for interchange with a client. Office 365 Office 365 mail servers (e.g. x-com.mail.protection.outlook.com) are always configured with the catch all policy, accepting all s sent to the domain and redirecting them to a central box for manual inspection. Catch all configured servers cannot respond to requests for address verification. This does not affect our coverage of Hotmail, Live and Outlook mail boxes. Punycode Punycode is a way to represent Unicode with the limited character subset of ASCII supported by the Domain Name System. See Wikipedia - Punycode for more information. RESTful Representational state transfer SMTP See Wikipedia - RESTful for further information. Simple Mail Transport Protocol SMTP is a protocol. It is the sequence of commands and responses between a client (the software sending an ) and server (the software receiving an ) that facilitates the sending and receiving of between computer based messaging systems. Spam Trap Spam traps are addresses used for the sole purpose of detecting spamming activities. Spam traps are used by many block lists (DNSBL) to detect spammers. For more information, see Wikipedia - Spam Traps. 20 Chapter 6. Glossary

25 Index B B2B, 19 B2C, 19 Block List, 19 D DDoS, 19 DEA, 19 DNS, 19 DNSBL, 19 E ESP, 20 G Grey Listing, 20 M MX, 20 O Office 365, 20 P Punycode, 20 R RESTful, 20 S SMTP, 20 Spam Trap, 20 21