one million mails a day: open source software to deal with it Charly Kühnast Municipal Datacenter for the Lower Rhine Area Moers, Germany
Internet many years ago... mail server mail client Seite: 2
today, however... Seite: 3
Part 1: spam filters Seite: 4
Spamfilters: First Contact Your Spamfilters should be the first to make contact with incoming SMTP connections. A well-configured spam filter will reduce 1,000,000 SMTP connections to 200,000. Seite: 5
Sicherheit von Internet-Anwendungen Spam Filter Header Checks RBLs Sender / Recipient Address Verification SMTPd PolicyD SpamAssassin to anti-virus server Seite: 6
Why should the spam filter be the first to receive new SMTP connections? Why not the anti-virus server or a mailrouting engine? Mar 15 12:12:23 spamfilter2 postfix/smtpd[22703]: connect from 82-142-adsl.vntc.ru[89.107.82.142] Mar 15 12:12:24 spamfilter2 postfix/smtpd[22703]: NOQUEUE: reject: RCPT from 82-142-adsl.vntc.ru[89.107.82.142]: 554 Service unavailable; Client host [89.107.82.142] blocked using cbl.abuseat.org Seite: 7
There are Real-Time Block Lists (RBLs) for - IP addresses and ranges known for sending spam - open mail relays - open proxies allowing the CONNECT method to SMTP - RfC ignorant servers -... Seite: 8
RBLs always a good thing? You have to trust other people's judgement about what is evil and what isn't When an RBL stops working, it might affect your mail transport On the other hand, the results are impressive. Seite: 9
Seite: 10 one million e-mails a day
Sender / Recipient Address Verification As soon as the SMTP dialogue reaches the rcpt to: stage, the spam filter checks with the originating server if the sender's address is actually valid. and / or with the destiantion server if the recipient's address is actually valid. Seite: 11
Sender / Recipient Address Verification Things that can go wrong: Catch-all configuration renders the verification useless When the probe connection is greylisted on the destination server, things usually go downhill. Seite: 12
policyd policyd integrates into your mail MTA and gives you White- Black- and Greylisting several Checks based on HELO data rate limiting Seite: 13
Greylisting Greylisting is still a sharp weapon when fighting spam (not as sharp as two years ago, though). However, Greylisting slows down mail delivery, so you're annoying both spammers and customers. Seite: 14
HELO, I'm a spammer policyd analyzes the HELO string and rejects the connection if your own hostname appears in the HELO string checks if an MTA rotates its HELO string with every mail it sends Seite: 15
Rate limiting many MTAs or, failing that, policyd can limit the rate of mails from a certain sender oder to a certain recipient within a given time frame. Seite: 16
SpamAssassin based on a ruleset, SpamAssassin assigns a Score to every Mail it processes. The Score can then be used to let the mail pass let the mail pass, but attach a warning tag (**spam?**) move the mail into a quarantine directory or drop it Seite: 17
Use OCR to get rid of image spam Spammers often conceal their offerings in pictures to evade anti-spam techniques. Tools like fuzzyocr, which integrates into SpamAssassin, extract the text from the images. Seite: 18
Part II: anti-virus server with AmavisD-new AmavisD-new is a Daemon written in Perl. It easily integrates in most MTAs. It is most commonly used to trigger anti-virus software and / or SpamAssassin MTA AmavisD-new Seite: 19
Scanning e-mail attachments for viruses is very time-consuming and likely to be the bottleneck in your mail transport chain. - use daemonized virus scanners - use very fast disk arrays or, even better, a SAN - let AmavisD use tmpfs! - configure syslog for asynchronous writing Seite: 20
Concurrency limits MTAs usually have reasonable concurrency limits to avoid overwhelming other mail servers. However, if your spam filter talks to you anti-virus server, the concurrency limit is likely to bog them down. Seite: 21
Part 3: Monitoring Of course you want to know what is going on on your mail servers. There are several nifty tools to visualize mail traffic, queues etc. Seite: 22
A classic: MRTG MRTG shows you the load on your mail server's network interface Seite: 23
Mailgraph mailgraph shows you how many mails were sent and received by this server Seite: 24
Mailgraph mailgraph also gives you statistics about bounced and rejected mails, as well as the amount of spam and viruses. Seite: 25
queuegraph Queuegraph tells you how many mails are in the active / deferred queue Seite: 26