EMB. Basics. Goals of this lab: Prerequisites: LXB, NET, DNS

Transcription

1 EMB Basics Goals of this lab: Learn how works under the hood Learn about spam filtering and virus scanning Gain experience setting up a full-fledged service Prerequisites: LXB, NET, DNS REVISION: 2.1 [ ] DAVID BYERS

2

3 Table of Contents PREPARATION... 1 Exercise 1: Review and preparation... 1 MAIN LAB... 3 Part 1: Basic mail server... 3 Exercise 2: Mail server installation and basic configuration... 4 Exercise 3: About the mail server... 4 DNS and mail service... 4 Exercise 4: Configure DNS for service... 4 Part 2: End user support... 5 Exercise 5: Implement IMAP server... 5 Expert options... 5 Part 3: Intermediate mail service... 5 User-friendly addresses... 6 Exercise 6: Configuring user-friendly addresses... 6 Spam protection... 6 Exercise 7: Questions about spam filtering... Error! Bookmark not defined. Exercise 8: Implementing spam filtering... 7 Virus filtering (optional)... 8 Exercise 9: Questions about server-based virus scanning... 8 Exercise 10: Implementation of virus scanning... 8 Expert options... 8 IDA/ADIT EMB: BASICS

4 IDA/ADIT EMB: BASICS

5 PREPARATION Do these exercises before proceeding to the main lab. You should be able to complete them using only the on-line Postfix documentation and a computer networks textbook. Exercise 1: Review and preparation Read the documentation for Postfix (or the mail server of your choice) and answer the following questions: 1-1 What processes does the mail server consist of? What does each of the processes do? 1-2 How can your mail server communicate with external programs that process mail (e.g. spam filters or virus scanners)? 1-3 What does a typical SMTP dialog look like? Show how to send a simple using the SMTP protocol. Answers to the questions above. Exercise 2: Spam protection 2-1 What type(s) of spam filtering does Spamassassin provide? 2-2 How does greylisting work? Under what circumstances could greylisting cause problems (e.g. non-acceptance of legitimate , or excessive delays)? 2-3 If users insist on challenge-response spam filtering, how can the third party impact be minimized (ideally nearly eliminated)? 2-4 What is SPF? What is DKIM? How do they prevent spam (and do they actually work)? Answers to all questions above. IDA/ADIT EMB: BASICS 1

6

7 MAIN LAB In this lab you will install and configure a complete mail system, complete from both a network and user point of view. Note that mail should work equally well on the clients as well as the server. A good test of the system is to send from a client to an external address and check that all headers in the are correct. Since you do not have access to a graphic display on the UML instances it is tricky to use graphical clients like Mozilla Thunderbird to read . An alternative that many seem to think works well is to use the text-base client mu t t. If you want to run software that relies on X from your UML instances you could also install necessary packages and login on the client from a lab workstation using ssh. Just make sure that X11 forwarding is enabled. Time taken 2006: 20-30h, average 25h Past problems: This lab has seen the widest variety of problems of any lab in the past, which is reflected in the large variation in time taken. Since the mail system is not only complex in itself, but also interfaces with nearly every other part of the system, there are lots of problem sources, and lots of ways to make mistakes. Therefore, it is vital to work professionally on this lab. That means understanding the software and its interactions with e.g. DNS, the network, the file system and the directory service; understanding the entire configuration and exactly what the role of every configuration statement is; and understanding how incoming and outgoing mail is processed, where files are stored and what is logged. It also means taking small, well-thought-out steps and confirming through testing that each step does exactly what it is supposed to do. You have to slow down if you want to reach the finish line as fast as possible. Part 1: Basic mail server There are lots of different mail servers available. In this course we recommend that beginners use the Postfix mail server. It is a high-quality production mail server that nevertheless is fairly easy to configure. More experienced students may want to try their hand at the classic sendmai l software. It is a bit more difficult to handle than Postfix, but infinitely more flexible. All Debian Gnu/Linux hosts must have a mail server. For hosts that will only forward all mail to another server, Postfix and sendmai l are overkill. Many Debian users like to use e x mh for such hosts, but an even simpler system, such as ssmt p will do the job equally well. You are strongly advised not to install qmai l unless you have previous experience with it. Before starting to install the mail server you must know what your goals are. Read the documentation first and prepare your installation in detail. Postfix documentation is available at and in the Postfix installation itself (man pages and example configuration files). Note that access to port 25 (SMTP) is filtered at the ingress to the university network. This means that you will not be able to send mail to your network from hosts outside the university. You can, however, send mail from your student account. Testing tip: When testing, try to keep the interactions between parts of the system to a minimum. For example, when testing if your server can receive mail, it is a good idea to start with a test case in which IDA/ADIT EMB: BASICS 3

8 you run SMTP manually using e.g. telnet to connect to the server. That way, you won t encounter problems with e.g. the mail user agent or DNS (which isn t completely configured for mail yet). Once you know raw SMTP is working, you should have a test case that involves a mail user agent. Similarly, when testing whether a client can send mail through your server, you should try manual SMTP first, perhaps followed by use of the sendmai l command on the client, and only when that works try using a MUA. Exercise 3: Mail server installation and basic configuration 3-1 Install the mail server of your choice. If you are unsure what to install, use Postfix. Your initial installation should meet the following requirements (minimum): (a) (b) (c) (d) Accept mail from any SMTP server in the world. Be able to send mail to any SMTP server in the world. Not accept mail for any other destinations than your domain. Meet the requirements of RFC 2821, section concerning the postmaster address. 3-2 Configure your clients to always use your mail server for outgoing mail. Client configuration is quite simple: they forward everything, untouched, to the central mail server. Clients must not accept connections from outside computers. Automated test cases that verify that your system meets the requirements. m When you have come this far you should understand how the server works. The following questions test your knowledge of key parts of the mail server. You should not proceed beyond this point before you can answer these questions, or you are likely to run into problems later. Exercise 4: About the mail server 4-1 Answer the following questions about your mail server: (a) (b) (c) (d) (e) (f) DNS and mail service What configuration files does the software use and what are they for? Which command displays the current queue of outgoing mail? Where is incoming stored before being delivered? Where is outgoing queued before delivery? Where does the mail server log messages about delivery? To where (directory and/or file) and in what format does the mail server deliver mail to end users? Answers to the questions above. For service to really work, you will need to configure DNS correctly. When an message is to be delivered, a DNS-aware mail server (which means pretty much every mail server currently running) will search for an MX record for the recipient s domain. MX records indicate hosts that can accept mail for a particular domain (known as Mail exchangers). Each domain may have several MXs, each with a priority. In your case there should be a single MX your mail server. Testing tip: When testing the following exercise, start by testing DNS alone, before you try delivering mail in a way that depends on DNS. That way, if you run in to problems with mail delivery, at least you know that DNS is functioning properly. Exercise 5: Configure DNS for service 5-1 Add the appropriate resource record(s) to your DNS server. No report required. IDA/ADIT EMB: BASICS 4

9 Part 2: End user support Sending mail isn t enough. Users tend to want to read mail as well (strange as that may sound). Today, the preferred way to read mail in a client-server setting is the IMAP protocol. You will implement a basic IMAP server on your system. The recommended IMAP server is Courier. However, Courier requires that your mail server delivers mail to users in so-called maildir format (the default delivery format is usually mbox format). If you are unwilling to reconfigure your mail server to deliver mail in maildir format, consider using uw-imapd as the IMAP server instead. You are advised not to use Cyrus. The Cyrus IMAP server is very capable, but its strengths are in environments with a high volume of mail and a very large number of users. It is not well suited to small installations. When installing the IMAP server, one of the things you need to check is that the IMAP server s idea of where and in what format user mailboxes are matches the mail server s. For example, if the IMAP server expects mail to be delivered in maildir format to a directory in the user s home directory, but the SMTP server delivers to mbox files in /var/mail, you won t be able to read mail using IMAP. Users must be able to read via IMAP from both your server and clients. Testing tip: As with SMTP, you should at least consider having a basic test case in which you run IMAP manually to ensure that you can distinguish failures in the IMAP software from other failures (e.g. networking, DNS or client configuration). You should also test IMAP with a real client, such as mutt or thunderbird (and these can be run on any computer on the Internet). Troubleshooting tip: In the past, a few groups have had a lot of problems with this part (most haven t). If you do run in to problems, you need to understand how IMAP works. That way, you can run a manual IMAP session (so you can see if the problems you have are in the client or on the server) and you can snoop the IMAP session (using wi r eshar k or by setting up a relay using socat ask your lab assistant for help) and understand what s going on. Exercise 6: Implement IMAP server 6-1 Install and configure and IMAP server that users on your clients and on the lab workstations can connect to, to read Where and in what format does your IMAP server expect user mailboxes to be? Automated test cases that verify that your system meets the requirements. If you elect to use Courier, then install the fam package first. Otherwise courier may not work correctly. Expert options If you want to try your hand at a more realistic implementation, configure SSL support in the IMAP server. Real sites will have SSL enabled to avoid sending passwords in the clear. If you want to test IMAP over SSL, you should be able to use socat to make the connection (rather than t el net ). If your IMAP server doesn t support SSL natively, then use st unnel as a frontend. Part 3: Intermediate mail service m Your mail service up to this point is technically adequate, but would not meet end-user demands. You still need some basic configuration and some add-on services. The clamav and spamassassin (spamd) processes (spam and virus filtering) use a lot of memory. Your server is configured to have 128Mb of memory by default, but the default configurations of these tools will almost certainly exhaust that. If you start running in to memory problems, stop the spamassassin and clamav processes, then configure the start scripts of spamassassin to start just one child process. If you still run in to problems, you may consider not running spamassassin and clamav as daemon processes, but start them on each incoming mail. IDA/ADIT EMB: BASICS 5

10 User-friendly addresses At this point, all outgoing mail probably has your mail server s fully qualified domain name as the sender domain. This is generally undesirable as you don t want to expose the mail server s name to end users. Furthermore, many organizations like outgoing mail to be stripped of internal user IDs. A common (but seriously flawed) system is to have outgoing mail look like it is sent from firstname.lastname@domain instead of userid@domain, which is what you would get with no special configuration. Transforming addresses like this is an example of address rewriting, a major function of all mail servers. Other examples of address rewriting include appending a domain name to sender addresses that don t have one and converting between different address formats. One thing to keep in mind is that rewriting rules are applied in a specific order, and may be very specific in what they rewrite. When working with address rewriting, make sure you know which order the various rules are applied in, and exactly to what, and how, each rule is applied. You will need to review both the Postfix manual (if you are using Postfix) and the manual pages for the various files involved. Exercise 7: Configuring user-friendly addresses 7-1 Configure your server so that all outgoing mail has your domain name, without any host name, to the right of sign, regardless of which machine the was sent from. No configuration is to be done on the clients; just on the server. 7-2 What are aliases and how are they configured? 7-3 Configure your server to accept mail to firstname.lastname@your-domain. Mail must be delivered to the appropriate user. There are at least three different ways of doing this some of which involve rewriting the address, and some of which that don t. Solutions that don t involve address rewriting are typically easier to work with. 7-4 What are sender canonical maps, and how are they configured? 7-5 Configure your server to rewrite outgoing mail addresses to firstname.lastname format automatically. For this to work, you may need to configure your clients to send all mail without the hostname, with only the hostname, or with something like localhost.localdomain after sign; the documentation on the address rewriting method you use explains why. 7-6 Formulate a policy for firstname.lastname addresses that is suitable for a large organization and that specifies how to handle naming conflicts in a consistent, fair and management-acceptable way. Spam protection Automated test cases that verify that your system meets the requirements. Every server today needs strong spam protection. Spam protection is a fairly hard problem. There are a number of concerns that must be dealt with: False positives False positives are non-spam messages that the system (incorrectly) detects as spam. False positives generally result in customers complaining about lost . False negatives False negatives are spam messages that are not detected as such. False negatives generally result in customers complaining about spam. Processing time Spam filtering can be resource intensive. A large mail system may need several dedicated high-end computers to do nothing but spam filtering. That generally results in complaints from management about costs. 3 rd party impact Spam filtering should never impact a third party (i.e. anyone other than the evil spammer and good-as-gold victim). I ll rant about that a little later. There are a number of different types of spam protection. Some types include content filters, DNSbased blacklists, challenge-response protocols and protocol hacks. IDA/ADIT EMB: BASICS 6

11 Content filters These tools look at the content of each message and use keywords, combinations of keywords, and other features to determine if a message is spam. Content filters can be static or dynamic. Static content filters are like expert systems, with a set of fixed rules that are applied to each message. Dynamic filters learn from example; each individual user must train the filter by indicating messages that are spam and messages that aren t. The filter will automatically learn to distinguish one from the other. Static content filters are fairly effective if the set of rules is updated frequently, and are suitable in installations with multiple users. Dynamic content filters are extremely effective, but each user must train the filter individually, or the number of false positives and false negatives will be too high. Static content filters are resource intensive. DNS-based blacklists DNS-based blacklists use special DNS servers to determine if the sending host is a legitimate sender. Blacklists can also be used to detect if a message has passed through a known open relay or otherwise insecure system (this is a good indication of spam). DNS-based blacklists are fast, but controversial as the policies of the publishers are not always clear and can be somewhat draconian. Challenge-response Challenge-response protocols are extremely effective at preventing spam, but they accomplish this by impacting third parties. Challenge-response systems will reply to messages from unknown senders with a challenge, which the sender must respond to (usually with another or by visiting a web site). Since most spammers use forged sender addresses, and often use valid addresses, challenges to spam messages tend to go to uninvolved third parties. In effect, users of challenge-response spam filters are making other people manually filter their spam for them. Protocol hacks Some newer methods of filtering use protocol hacks to determine if a message is spam or not. For example, SPF (Sender Protected Framework) uses DNS records to indicate valid originating mail servers for a domain (thus making it harder to forge senders in that domain), and Greylisting uses the SMTP protocol to implement what nearly amounts to a challenge-response protocol without the third party impact. Other possibilities also exist, but few are in widespread deployment. Nolisting exploits how DNS lookups for servers are supposed to be done. Exercise 8: Implementing spam filtering 8-1 Configure your mail server so Spamassassin can examine each message before it is delivered. Spam should be marked as such, but not bounced or deleted. Use spamc/spamd. 8-2 Implement greylisting in your mail server. Ensure that local clients are not subjected to greylisting. 8-3 Configure your DNS server to include SPF records for your domain. The only acceptable as a source of from your domain is your server. The SPF record should reflect this. Automated test cases that verify that your system meets the requirements. Testing tip: When testing spam filtering you need a message that will be categorized as spam. Most spam filters will react to the Generic Test for Unsolicited Bulk test string. See for more information. Testing tip: Your tests for greylisting should include manual SMTP sessions. In fact, this is the easiest way to test that greylisting works, once you understand how it is supposed to work. This is also a good time to do some regression testing to ensure that your own clients can still send mail without it being greylisted. If you find that your clients are unable to send mail (or are subject to greylisting) at this point, you probably need to review the configuration changes you made, to ensure that you are aware of all the implications of the changes. Testing tip: Testing SPF requires either a testing service or a mail server that uses SPF records. If you have access to a GMail account, you can view all the headers of mail sent mail to it from your system IDA/ADIT EMB: BASICS 7

12 and see how it reacts to your SPF records. You can also use the on-line SPF tester at Virus filtering (optional) This section is optional. If you want to try your hand at setting up a more complete service, then proceed, but if you ve had enough of configuration, stop. Your mail server should have good protection against viruses. Since many viruses are delivered by today it has become common practice to implement server-side virus protection, even on Unix type mail systems. Today, there are a great number of both commercial and free alternatives to choose from. Exercise 9: Questions about server-based virus scanning 9-1 What is the most important benefit of server-based virus scanning over client-based virus scanning (where each client scans for viruses in incoming messages)? Answers to the questions above. Regrettably, many implementations of server-based virus filters have a high impact on third parties. For various reasons, server-based virus scanning is often done after the SMTP dialog is completed (scanning is resource intensive, and some antivirus products do not integrate well with mail servers). Many antivirus products are configured by default to send a reply to the originator of the virus-infected . Unfortunately, this reply is hardly ever seen by the real sender (whose host is infected) since practically every virus forges the sender address. Exercise 10: Implementation of virus scanning 10-1 Implement server-based virus scanning on your mail server. Use ClamAV for virus scanning. If a virus is detected, you should not reply to the sender. It is acceptable to either notify the recipient that a virus has been found, or generate an SMTP error message (this assumes that scanning takes place during the SMTP dialog). Automated test cases that verify that your system meets the requirements. Combining virus filtering and spam filtering with amavisd-new Amavisd-new is free software which can be used to connect an server with other programs, and it is quite simple to set up. One problem with amavisd-new is that it prevents spamassassin from rewriting headers, so if you choose to use amavisd-new, you need to think hard about how to make spam filtering really work (and how to demonstrate it). If you are planning to use amavisd-new together with Postfix, LMTP is the preferred solution. Expert options If you want to try your hand at some more advanced SMTP configuration, you could try implementing support for SSL (encrypted connections) either using a dedicated port for SMTP over SSL or by supporting STARTTLS. You can also implement SMTP authentication, so that users can use your mail server to send mail even if they are not on your network, as long as they can authenticate. If you want to test SMTP over SSL (not sure if this can be made to work with STARTTLS) you should be able to use socat to make the connection (rather than t el net ). IDA/ADIT EMB: BASICS 8

13 FEEDBACK FORM EMB Complete this feedback form individually at the end of the lab and hand it to the lab assistant when you finish. Your feedback is essential for improving the labs. Each student should hand in a feedback form. Do not cooperate on completing the form. You do not need to put your name on the feedback form. Your feedback will be evaluated the same way regardless of whether your name is on it or not. Your name is valuable to us in case you have made and comments in the last section that need clarifications or otherwise warrant a follow-up. For each section, please rate the following (range 1 to 5 in all cases). Difficulty: Rate the degree of difficulty (1=too easy, 5=too difficult) Learning: Rate your learning experience (1=learned nothing, 5=learned a lot). Interest: Rate your interest level after completing the part (1=no interest, 5=high interest). Time: How long did the part take to complete (in minutes)? Difficulty Learning Interest Time (minutes) Preparation Part 1: Basic mail server Part 2: End user support Part 3: Intermediate mail service Overall Please answer the following questions: What did you like about this lab? What did you dislike about this lab? Make a suggestion to improve this lab. IDA/ADIT EMB: BASICS

14

15 FEEDBACK FORM EMB Complete this feedback form individually at the end of the lab and hand it to the lab assistant when you finish. Your feedback is essential for improving the labs. Each student should hand in a feedback form. Do not cooperate on completing the form. You do not need to put your name on the feedback form. Your feedback will be evaluated the same way regardless of whether your name is on it or not. Your name is valuable to us in case you have made and comments in the last section that need clarifications or otherwise warrant a follow-up. For each section, please rate the following (range 1 to 5 in all cases). Difficulty: Rate the degree of difficulty (1=too easy, 5=too difficult) Learning: Rate your learning experience (1=learned nothing, 5=learned a lot). Interest: Rate your interest level after completing the part (1=no interest, 5=high interest). Time: How long did the part take to complete (in minutes)? Difficulty Learning Interest Time (minutes) Preparation Part 1: Basic mail server Part 2: End user support Part 3: Intermediate mail service Overall Please answer the following questions: What did you like about this lab? What did you dislike about this lab? Make a suggestion to improve this lab. IDA/ADIT EMB: BASICS