Email Basics: Email protocols were developed even before there was an Internet, at a time when no one was anticipating widespread use of digital graphics or even rich text format (fonts, colors, etc.), not to mention languages that use other than the Roman alphabet. So email only handled ASCII messages. The main protocols for email are: SMTP Simple Mail Transfer Protocol outgoing mail and relaying POP3 Post Office Protocol 3 incoming mail box IMAP Internet Mail Access Protocol incoming mail box MIME Multipurpose Internet Mail Extensions Every email is sent out by an SMTP server. Incoming mail boxes exist on either a POP3 or an IMAP server. These 2 protocols are very similar, the differences being mostly in the defaults for downloading or keeping copies of received emails when the account owner logs in to check his/her email. IMAP is the more modern server and is most likely to be used. MIME is the protocol that was devised to allow non-ascii encoded content in an email and attached files to an email. GENERAL OPERATION OF EMAIL There are two ways in which email can be originated and received. The difference between these two is very important to understand: 1. Traditional email requires you to use email client software on your computer (Outlook, Outlook Express, Apple Mail, Thunderbird), which is configured to communicate with a specific SMTP server which you are authorized to use to send out mail, and a specific IMAP (or POP3) server where you have an email account and INBOX. These servers are normally provided for your use by your Internet Service Provider (ISP). 2. Webmail such as Gmail, AOL, Yahoo mail, Hotmail, etc. only require you to use a browser (Internet Explorer, Firefox, Safari, etc. or on a mobile device) because the equivalent of an email client is part of the website where you have an account. Once a message has been sent from an SMTP server, it is routed to its destination typically through several mail transfer agents (or relay servers), regardless of whether it was created with Traditional or Webmail software. Depending on whether an email recipient uses Webmail or Traditional software, the destination INBOX receives the message and stores it until the recipient logs into their account to read their new (or saved) email.
As a functional overview, you may think of email transmission as follows: Some means for originator to compose an email message Some means to route messages to their destinations Some means to store messages for a recipient (INBOX) until they call for them Some means for recipient to recover messages and store them, once read Traditional Email client (MUA) on originator s computer Webmail Website, originator only needs a web browser Mail Transfer Agents (MTA) or mail relay servers IMAP or POP3 server at your ISP, from where you can download them to your computer Email client (MUA) on recipient s computer IMAP server at website where you have an account (you can also download them to your computer) Website, recipient only needs a web browser * Email is based around the use of electronic mailboxes. When an email is sent, the message is routed from server to server, all the way to the recipient's email server. More precisely, the message is sent to the mail server tasked with transporting emails (called the MTA, for Mail Transport Agent) to the recipient's MTA. On the Internet, MTAs communicate with one another using the protocol SMTP, and so are logically called SMTP servers (or sometimes outgoing mail servers). The recipient's MTA then delivers the email to the incoming mail server (called the MDA, for Mail Delivery Agent), which stores the email as it waits for the user to accept it.
To use a real-world analogy, MTAs act as the post office (the sorting area and mail carrier, which handle message transportation), while MDAs act as mailboxes, which store messages (as much as their volume will allow) until the recipients check the box. This means that it is not necessary for recipients to be connected in order for them to be sent email. To keep everyone from checking other users' emails, MDA is protected by a user name called a login and by a password. Retrieving mail is done using a software program called an MUA (Mail User Agent). When the MUA is a program installed on the user's system, it is called an email client (such as Mozilla Thunderbird, Microsoft Outlook, Eudora Mail, etc.). When it is a web interface used for interacting with the incoming mail server, it is called webmail. * From: http://en.kioskea.net/contents/courrier-electronique/fonctionnement-mta-mua.php3 Anatomy of an Email We are all familiar with what an email looks like when we view one on our computer or mobile device screen, but that s only what it looks like after it has been processed for display on that device or with that software. There are many header records that are not displayed, by default, but you can view them by selecting View Source or similar wording, from your mail client screen. These are headers that provide instructions to your mail client, or which document the processing that has occurred on each SMTP server that has handled the message; a precise time-line and identification of each server, special information such as spam filtering that may have occurred, etc. Ordinarily, you won t be interested in seeing such headers, so they are not normally displayed; but if you want to trace a particular message, perhaps to see if it actually came from who the visible header SAYS it comes from, you can do so. Then there is all the encoded data that is made possible by the MIME protocol, including HTML content, rich text format, inline images, or even encrypted text, if you have received encrypted email. All of these forms of content would have to be encoded at the originator s SMTP server into ASCII characters, for transmission as email, and finally decoded by your email client of MUA (Mail User Agent) in order for you to be able to read it. Finally, there may be attached files to an email message. These are commonly shown at the bottom of the main body of an email message, they often require software to display them, such as Adobe Reader if the attachment is a.pdf file, or Windows Media Player if the attachment is a video clip, or Microsoft Excel if the
attachment is a spreadsheet, etc., and may also be copied as files to your computer or mobile device. MIME protocol defines multipart messages, meaning that a single email message may consist of several parts, for example, a plain ASCII text body, which might also include an inline encoded image, as well as an attached MS Word document; and all of these would be separated by unique boundary markers. I sent myself a couple of test emails so I could demonstrate the various components of a MIME email. This is what actually gets transferred from the originator s MUA through all the MTA s to the recipient s MUA. First, a very simple email with no MIME content, addressed to just one person. Looking at it as it is displayed in my Thunderbird email client: Notice that I clicked on the Other Action dropdown list, where View Source can be found in Thunderbird. In other email clients, the selection may be called View Fulll Headers and be found in a different location.
But the actual content of the email was this: From - Fri Nov 09 20:28:24 2012 X-Account-Key: account2 X-UIDL: 1352521701.726584.p3plgemini06-04.prod.phx.1091074368 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: Received: (qmail 7217 invoked by uid 30297); 10 Nov 2012 04:28:21-0000 Received: from unknown (HELO p3pismtp01-071.prod.phx3.secureserver.net) ([10.6.12.190]) (envelope-sender <don94403+caf_=don=ravey.net@gmail.com>) by p3plsmtp06-03.prod.phx3.secureserver.net (qmail-1.03) with SMTP for <don@ravey.net>; 10 Nov 2012 04:28:21-0000 X-IronPort-Anti-Spam-Result: ArsFAPLUnVDRVdgvk2dsb2JhbABEwkwCAgJ4CCMBAQEBCQkLCRQEI4I+H0AqExYYAwIBAgEiAQUBIg0IAQG IBpxCgm+eN4xqhXQDiFqNIYVriHc/hDKBQw Received: from mail-qa0-f47.google.com ([209.85.216.47]) by p3pismtp01-071.prod.phx3.secureserver.net with ESMTP; 09 Nov 2012 21:27:23-0700 Received: by mail-qa0-f47.google.com with SMTP id t11so869453qaa.13 for <don@ravey.net>; Fri, 09 Nov 2012 20:27:23-0800 (PST) Received: by 10.49.49.167 with SMTP id v7mr21803027qen.65.1352521642973; Fri, 09 Nov 2012 20:27:22-0800 (PST) X-Forwarded-To: don@ravey.net X-Forwarded-For: don94403@gmail.com don@ravey.net Delivered-To: don94403@gmail.com Received: by 10.49.64.72 with SMTP id m8csp122705qes; Fri, 9 Nov 2012 20:27:22-0800 (PST) Received: by 10.68.244.135 with SMTP id xg7mr39389854pbc.87.1352521641911; Fri, 09 Nov 2012 20:27:21-0800 (PST) Return-Path: <don@ravey.net> Received: from p3plsmtpa08-08.prod.phx3.secureserver.net (p3plsmtpa08-08.prod.phx3.secureserver.net. [173.201.193.109]) by mx.google.com with ESMTP id bt8si626580pab.100.2012.11.09.20.27.21; Fri, 09 Nov 2012 20:27:21-0800 (PST) Authentication-Results: mx.google.com; spf=pass (google.com: domain of don@ravey.net designates 173.201.193.109 as permitted sender) smtp.mail=don@ravey.net Received: from [192.168.1.4] ([173.228.16.19]) by p3plsmtpa08-08.prod.phx3.secureserver.net with id MgTL1k0020Qgtx101gTLnL; Fri, 09 Nov 2012 21:27:21-0700 Message-ID: <509DD7AA.1080906@ravey.net> Date: Fri, 09 Nov 2012 20:27:22-0800 From: Don Ravey <don@ravey.net> User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Donald Ravey <don94403@gmail.com> Subject: Simple test email Content-Type: text/plain; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Nonspam: None X-Antivirus: AVG for E-mail 2012.0.2221 [2441/5385] X-AVG-ID: ID768C36C1-7B9BE208 Hello, This is about the simplest email possible. Don
I highlighted in green the lines that are shown in the normal display of the email. You can see that there is a great deal more information in the headers than is displayed by default. I highlighted in red some of the more useful header lines. Each MTA (mail relay) server that handles a message adds a Received: line to the top of the headers, so this message sent to my Gmail address (which I have configured to forward all incoming mail to my ravey.net Inbox) actually went through 8 MTA servers! By comparing the timestamps on the Received: lines, you can trace exactly how long it took to get from my GoDaddy SMTP server (secureserver.net) through several MTA s and was delivered to Google s IMAP server (gmail.com), then forwarded back to GoDaddy s SMTP servers to its IMAP server, from where I recovered it with my Thunderbird email client on my computer, less than a minute later! But that was a simple plain text email. Then I sent another email that included some HTML content, a small.ico inline embedded image (the clock), and had a small.xls Excel spreadsheet file attached to the email. This is what that looked like in my Thunderbird client:
Notice the indication of an attached spreadsheet file at the bottom. I could double-click on that to open the file with Microsoft Excel, or I could opt to save the file on my hard drive. But things like HTML, the inline image, and the attached spreadsheet file are NOT plain ASCII text, so those parts must be encoded (converted to ASCII characters) and MIME instructions added to the message. Here is the source code of that message, showing all the headers, as with the first message, and all the MIME instructions and encoded content (well, not ALL of it, that would have taken pages and pages! I have indicated where I have skipped most of the actual encoding. From - Fri Nov 09 22:11:15 2012 X-Account-Key: account2 X-UIDL: 1352527860.273414.p3plgemini06-01.prod.phx.1113516352 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: Received: (qmail 30211 invoked by uid 30297); 10 Nov 2012 06:11:00-0000 Received: from unknown (HELO p3pismtp01-067.prod.phx3.secureserver.net) ([10.6.12.186]) (envelope-sender <don94403+caf_=don=ravey.net@gmail.com>) by p3plsmtp06-01.prod.phx3.secureserver.net (qmail-1.03) with SMTP for <don@ravey.net>; 10 Nov 2012 06:11:00-0000 X-IronPort-Anti-Spam-Result: An4CAEbunVDRVdg2k2dsb2JhbABEgksDgwSGJrZoawgjAQEBAQkJCwkUBCOCFQoBBUUzARAMAQEBCg8BAgE WDwIHAwIBAgEUDgEFARIKBg0BBAECAQGIBp80ni+SXgOIWoYhgSEBhV6GDYhVP4QygUM Received: from mail-qa0-f54.google.com ([209.85.216.54]) by p3pismtp01-067.prod.phx3.secureserver.net with ESMTP; 09 Nov 2012 23:10:59-0700 Received: by mail-qa0-f54.google.com with SMTP id c10so923147qae.13 for <don@ravey.net>; Fri, 09 Nov 2012 22:10:59-0800 (PST) Received: by 10.49.3.102 with SMTP id b6mr10277806qeb.26.1352527859148; Fri, 09 Nov 2012 22:10:59-0800 (PST) X-Forwarded-To: don@ravey.net X-Forwarded-For: don94403@gmail.com don@ravey.net Delivered-To: don94403@gmail.com Received: by 10.49.64.72 with SMTP id m8csp128503qes; Fri, 9 Nov 2012 22:10:57-0800 (PST) Received: by 10.68.135.42 with SMTP id pp10mr40432503pbb.159.1352527856772; Fri, 09 Nov 2012 22:10:56-0800 (PST) Return-Path: <don@ravey.net> Received: from p3plsmtpa01-02.prod.phx3.secureserver.net (p3plsmtpa01-02.prod.phx3.secureserver.net. [72.167.82.82]) by mx.google.com with SMTP id wu8si851881pbc.333.2012.11.09.22.10.55; Fri, 09 Nov 2012 22:10:56-0800 (PST) Authentication-Results: mx.google.com; spf=pass (google.com: domain of don@ravey.net designates 72.167.82.82 as permitted sender) smtp.mail=don@ravey.net Received: (qmail 9878 invoked from network); 10 Nov 2012 06:10:55-0000 Received: from unknown (173.228.16.19) by p3plsmtpa01-02.prod.phx3.secureserver.net (72.167.82.82) with ESMTP; 10 Nov 2012 06:10:55-0000 Message-ID: <509DEFF0.3070006@ravey.net> Date: Fri, 09 Nov 2012 22:10:56-0800 From: Don Ravey <don@ravey.net> User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Donald Ravey <don94403@gmail.com>
CC: don@ravey.net Subject: Fwd: More complex email test References: <509DEEFB.8010109@ravey.net> In-Reply-To: <509DEEFB.8010109@ravey.net> X-Forwarded-Message-Id: <509DEEFB.8010109@ravey.net> Content-Type: multipart/mixed; boundary="------------070708080104040101020206" X-Nonspam: None X-Antivirus: AVG for E-mail 2012.0.2221 [2441/5385] X-AVG-ID: ID1D3332C7-5BB931C7 This is a multi-part message in MIME format. --------------070708080104040101020206 Content-Type: multipart/alternative; boundary="------------040307070504000906040102" --------------040307070504000906040102 Content-Type: text/plain; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: 7bit OK, now this part is just plain ASCII text....and this part is HTML styled text...and this is a small, encoded, embedded inline image: clock...and I will attach a small spreadsheet file to this email. Don --------------040307070504000906040102 Content-Type: multipart/related; boundary="------------080005050403070008080401" --------------080005050403070008080401 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=iso-8859-1"> </head> <body text="#000000" bgcolor="#ffffff"> <div class="moz-forward-container"> <meta http-equiv="content-type" content="text/html; charset=iso-8859-1"> OK, now this part is just plain ASCII text. <div style="background-color:#ccf; color:#069;">
...and this part is HTML styled text </div>...and this is a small, encoded, embedded inline image: <img alt="clock" src="cid:part1.05040107.09050607@ravey.net" height="48" width="48">...and I will attach a small spreadsheet file to this email. Don </div> </body> </html> --------------080005050403070008080401 Content-Type: image/x-icon Content-Transfer-Encoding: base64 Content-ID: <part1.05040107.09050607@ravey.net> AAABAAEAMDAAAAEAGACoHAAAFgAAACgAAAAwAAAAYAAAAAEAGAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmQAAmQAAmQAAmQAAmQAA mqaamqaamqaamqaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa AAAAAAAAAAAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAA mqaamqaamqaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmQAAmQAAmQAAmQAA mqaamqd///////////////////////////////////8amqaamqaamqaamqaamqaaaaaaaaaa AAAAAAAAAAAAAAAAAAAAAAAAmQAAmQAAmQAAmQD///////////////////////////8AAAAA mqaamqd///////////////////////8amqaamqaamqaa... I omitted about a page and a half here... AAAAAAAAAAAAAAAAmQAAmQAAmQAAmQAAmQAA mqaamqaamqaamqaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
AAAAAAAAAAD////////8/f////////z9/////////P3////////8/f////////z9//////// /P3////////8/f//AH////z9//AAA////P3/4AAB///8/f+ACAB///z9/wARAD///P38ABEA D//8/fwAEAAP//z9+AAAAAf//P3wAAQAA//8/fAAAAAD//z94AAAAAH//P3AAAAAAP/8/cAA AAAA//z9wAAAAAD//P3AAAAAAP/8/YAAAAAAf/z9gAAAAAB//P2AAAAAIH/8/YAAAAAAf/z9 gaaaacb//p2aaaaaah/8/yaaaaagf/z9gaaaaab//p2haaaaah/8/yaaaaaaf/z9waaaaad/ /P3AAAAAAP/8/cAAAAAA//z9wAAAAAD//P3gAAAAAf/8/fAAAAAD//z98AAAAAP//P34AAAA B//8/fwAAwAP//z9/gACAB///P3/AAEAP//8/f+AAAB///z9/+AAAf///P3/8AAD///8/f// AH////z9/////////P0= --------------080005050403070008080401-- --------------040307070504000906040102-- --------------070708080104040101020206 Content-Type: application/excel; name="test.xls" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="test.xls" 0M8R4KGxGuEAAAAAAAAAAAAAAAAAAAAAPgADAP7/CQAGAAAAAAAAAAAAAAABAAAAGQAAAAAA AAAAEAAA/v///wAAAAD+////AAAAABgAAAD///////////////////////////////////// //////////////////////////////////8JCBAAAAYFAJogzQfBwAAABgMAAOEAAgCwBMEA AgAAAOIAAABcAHAADAAARG9uYWxkIFJhdmV5ICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIEIAAgCwBGEBAgAAAMABAAA9AQYAAQACAAMAnAACAA4AGQACAAAA EgACAAAAEwACAAAArwECAAAAvAECAAAAPQASAHgAhwC5VUwsOAAAAAAAAQBYAkAAAgAAAI0A AgAAACIAAgAAAA4AAgABALcBAgAAANoAAg... And here I omitted another 4 pages or so... 1IAbwBvAHQAIABFAG4AdAByAHkAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAWAAUB//////////8CAAAA IAgCAAAAAADAAAAAAAAARgAAAAAAAAAAAAAAAAAAAAAAAAAA/v///wAAAAAAAAAAVwBvAHIA awbiag8abwbraaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa AAAAABIAAgH///////////////8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAABAAAAAAAAAFAFMAdQBtAG0AYQByAHkASQBuAGYAbwByAG0AYQB0AGkAbwBuAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKAACAQEAAAADAAAA/////wAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAAAAEAAAAAAAAAUARABvAGMAdQBtAGUAbgB0AFMA dqbtag0ayqbyahkasqbuagyabwbyag0ayqb0agkabwbuaaaaaaaaaaaaaaa4aaib//////// ////////AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAQAAAAAAAA --------------070708080104040101020206-- In addition to the highlighting I used in the first email example, here I added brown highlighting for MIME instructions, such as the unique random generated boundary markers, and violet highlighting for the encoded image and spreadsheet data. So this should give you an idea of what is actually contained in your emails as they are sent through various routing servers and finally delivered to somebody s INBOX, awaiting their logging in to view their new emails! It involves much more than the average Internet user realizes! DLR 11/10/2012