Controlling mail message flow by content analysis

Transcription

1 Controlling mail message flow by content analysis Ralf Hildebrandt Patrick Ben Koetter Open Source Days 2008 Copenhagen, 04. October 2008

2 1 Stages of spam-prevention Where can spam be controlled? Split anti spam measures SMTP Session Control Content Control 2 Integrating amavid-new into Postfix smtpd_proxy_filter content_filter Where is the difference? 3 The nature of content control Risks Requirements 4 amavisd-new History Architecture

3 5 Implementations Typical Mailserver Typical Mailgateway A dedicated Scanhost 6 Basic Setup Logging Virus Scanners Spam Scanners Quarantine Notification 7 Housekeeping Quarantine Management Quarantine Notification Quarantine Release Log analysis and statistics

4 8 Individualization Maps LDAP Policy Banks Policy Banks Example 9 Beyond the standard stuff Keeping SpamAssassin up to date Extending SpamAssassins rules Extending ClamAV patterns Pen-Pals p0f - passive operating system fingerprinting DKIM - Domainkeys Identified Mail

5 Scan before and after queue can be scanned before and after queueing

6 Split anti spam measures Measures are typically split

7 A day in the life of a mail server More than 98% of mail is spam.

8 A not so good idea... Content scanning is resource expensive. Don t let the full load hit the content scanner

9 SMTP Session Control reduces load Reject early! Use SMTP Session Control to minimize the mail that hits after queue measures

10 Limitations of Before-Queue-Scanning In a perfect world content inspection takes place before queuing, but clients may run into a timeout.

11 SMTP Session Control Best practices require RFC compliance from your senders require RFC compliance first kill spam later order restrictions from cheap to expensive allow for exceptions!

12 smtpd_recipient_restrictions I Cheap restrictions: smtpd_recipient_restrictions = reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, check_client_access cdb:/etc/postfix/mynetworks permit_sasl_authenticated, reject_unauth_destination, reject_unlisted_recipient,... These keep your users from sending the worst nonsense.

13 smtpd_recipient_restrictions II Cheap restrictions:... check_helo_access pcre:/etc/postfix/helo_checks.pcre, reject_invalid_helo_hostname reject_non_fqdn_helo_hostname check_recipient_access cdb:/etc/postfix/roleaccount_exceptions, check_sender_access cdb:/etc/postfix/sender_checks check_sender_access pcre:/etc/postfix/sender_checks.pcre... We re checking the HELO for obvious signs of forgery and block some senders.

14 smtpd_recipient_restrictions III Expensive restrictions:... check_client_access check_sender_mx_access check_sender_ns_access check_sender_mx_access check_sender_mx_access check_sender_mx_access check_client_access cidr:/etc/postfix/drop.cidr cidr:/etc/postfix/drop.cidr cidr:/etc/postfix/drop.cidr cidr:/etc/postfix/bogon_networks.cidr, cidr:/etc/postfix/wildcard_mx_records.cidr, cdb:/etc/postfix/mx_accepts_no_bounces cdb:/etc/postfix/client_checks The checks using cidr:/etc/postfix/drop.cidr check if a sender domain s MX or NS record point to networks controlled by spammers. This cidr:/etc/postfix/bogon_networks.cidr check if a sender domain s MX record points to private netspace....

15 smtpd_recipient_restrictions IV Expensive restrictions:... reject_unknown_reverse_client_hostname reject_rbl_client zen.spamhaus.org reject_rbl_client bl.spamcop.net reject_rbl_client b.barracudacentral.org check_sender_access cdb:/etc/postfix/rhsbl_exceptions reject_rhsbl_sender dsn.rfc-ignorant.org

16 Consequences Postfix rejects a lot of connections early content_filter and smtpd_proxy_filter are relieved from a lot of load (90% rejection rate)

17 header_checks header_checks check headers only: /^Received: from.*by nomail\.charite\.de/ REJECT Fake nomail.charite.de Received: Header found /Date:.*([3-9].:..: 2[4-9]:..: :[6-9].: :..:6[1-9])/ REJECT Invalid time "${1}" in Date header

18 mime_header_checks mime_header_checks check MIME-headers only: /name=\"(.*)\.(386 bat chm cmd com do exe hta jse lnk msi)\"$/ REJECT Unwanted attachment $1.$2

19 body_checks body_checks checks everything that s not a header: if /^Received: from.*by mail(-ausfall)?\.charite\.de/!/ \(Postfix\) with /i REJECT Fake charite.de Received: Header found, this \ is a bounce for a mail our system did not send! endif

20 smtpd_proxy_filter Accept mail from the internet, pass it on to the filter on [ ]:10025 Caution Enable XFORWARD this allows amavisd-new to see the actual client instead of just smtp inet n smtpd -o receive_override_options=no_address_mappings -o smtpd_authorized_xforward_hosts= /8 -o smtpd_authorized_xclient_hosts= /8 -o smtpd_proxy_filter=[ ]:10025

21 Reinjection Accept the mail back into Postfix after amavisd-new is done with it: Caution Disable filter on reentry! localhost:10026 inet n smtpd -o smtpd_proxy_filter= -o content_filter= -o receive_override_options=no_unknown_recipient_checks -o smtpd_authorized_xforward_hosts= /8 -o smtpd_authorized_xclient_hosts= /8 -o mynetworks= /8 -o smtpd_recipient_restrictions=permit_mynetworks,reject -o syslog_name=reinjection

22 content_filter Accept mail from the internet, pass it on to the filter on [ ]:10025 Caution Enable XFORWARD this allows amavisd-new to see the actual client instead of just smtp inet n smtpd -o receive_override_options=no_address_mappings -o smtpd_authorized_xforward_hosts= /8 -o smtpd_authorized_xclient_hosts= /8 -o content_filter=smtp:[ ]:10025

23 Where is the difference? Caution with smtpd_proxy_filter the mail is processed as it arrives ( just in time -scanning) with content_filter the mail is queued first and scanned later Clients may not be patient enough to wait until your system is willing to accept and process mail.

24 The nature of content control

25 Risks Content contains malware Content may decompress without bounds (compression bombs) Content cannot be scanned (encrypted) Content cannot be categorized

26 Requirements Unpack into sandbox Apply limits to size(s) Apply warnings to messages

27 A brief history Shell script that pipes mail to a scanner (1997) First Perl program (2000) Perl daemon (2001) Modular Design (2002) Perl daemon, pre-fork, NET::Server ( ) by Marc Martinec Add internal Anti Spam methods (2006) Add p0f, pen-pals (2007) Add DKIM (2008)

28 Internal Process Chain is processed in several stages

29 Decomposing Save resources! Decompose once, analyze multiple times Configuration settings $MAXLEVELS $MAXFILES $MIN_EXPANSION_QUOTA $MAX_EXPANSION_QUOTA

30 Adding Information p0f results DKIM signature verification results XFORWARD data

31 Content Inspection SpamAssassin is the default Anti-Spam tool Adding dspam is optional ClamAV is the default Anti-Virus tool Internal Methods Inspect MIME types Check if suffix is black-/whitelisted

32 Content Classification CLEAN SPAMMY SPAM VIRUS BANNED BADH OVERSIZED MTA

33 Actions Store (bad) message in quarantine Archive any message that flows through the system Reinject message into SMTP transport Notify sender/recipient about the message status D_REJECT a message D_DISCARD a message D_BOUNCE a message D_PASS a message

34 DKIM Signing... a domain-level authentication framework for using public-key cryptography and key server technology to permit a signing domain to assert responsibility for a message, thus protecting message signer identity and the integrity of the messages... Generate private key and public certificate Provide a per-domain/per-user list of DKIM signature options Add DKIM signatures to outgoing messages

35 Typical Mailserver Everything takes place on one machine.

36 Typical Mailgateway The gateway examines content. Follow-Up systems only do SMTP session control.

37 A dedicated Scanhost Gateways do SMTP session control. Content control is done by a dedicated Content Inspection server.

38 Logging syslog or native loglevel from silent to go buy new harddisks

39 Virus Scanners primary scanners are/should be daemons secondary scanners are scripts They are called only if primary scanners are not available Both groups allow to define a chain of defining ClamAV access parameters [ ClamAV-clamd, \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.ctl"], qr/\bok$/, qr/\bfound$/, qr/^.*?: (?!Infected Archive)(.*) FOUND$/ ]

40 Spam Scanners amavisd-new uses Mail::SpamAssassin as default spam scanner dspam can be used as an alternative amavisd-new only configures/controls a few SpamAssassin parameters Additional SpamAssassin configuration makes SA more efficient Caution amavisd-new does not use spamc or spamd!

41 Quarantine Quarantine destinations are send message to a mailbox store message in a file store message in a SQL server

42 Notification sender and recipients can be notified about a messages status notifications can be enabled on a global, a per-domain or on a per-user basis transport method can be set separately from the regular $forward_method powerful macros and templates customize notification messages

43 Quarantine Management amavisd-release releases messages from quarantine Messages can be sent as attachments amavisd-release is not enabled by default Enabling amavisd-release in amavisd.conf $interface_policy{ SOCK } = AM.PDP ; $policy_bank{ AM.PDP } = {protocol=> AM.PDP }; $unix_socketname= /var/amavis/amavisd.sock ;

44 Quarantine Notification

45 Releasing a message from Quarantine p@mail:~$ sudo amavisd-release virus-monrpcyyy3wo [sudo] password for p: Ok: queued as A974B820007

46 Viewing a released message

47 Log analysis and statistics Know your enemy! mailgraph overall picture at a glance amavisd-nanny gives you info about what the processes are doing amavisd-agent gives SNMP-style statistics amavisd-logwatch catch trends, document daily work manual inspection inspect manually

48 amavisd-nanny

49 mailgraph

50 mailgraph II

51 amavisd-agent

52 amavisd-logwatch

53 Individualization I Problem amavisd-new carries lots of legacy code! amavisd-new started as a single user script multi user environment requires individual settings newer groups of parameters wrap around older groups like layers of onion skin Map Types Single setting Lists Regular Expression

54 Individualization II Layers of onion skin # Check for = ( \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re ); # Check for = ( \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re );

55 Maps amavisd-new has a map for nearly @virus_name_to_spam_score_maps...

56 Maps Uses for maps domains create a list of domains amavisd is responsible for exclusion exclude someone from a check exception deliver mail even if it is banned

57 Maps: Domains Postfix and amavisd-new share = ( \@local_domains_acl, read_hash( /etc/postfix/virtual_mailbox_domains );

58 Maps: Exclusion Exclude a user from spam = ( ["user\@example.com"] );

59 Maps: Domains Allow a user to receive banned = ( ["user\@example.com"], );

60 LDAP I Benefits Enterprise ready Single configuration backend Simplify user administration Fast read access

61 LDAP II Enabling and configuring LDAP lookups in amavisd-new # Enable LDAP $enable_ldap = 1; # LDAP access $default_ldap = { hostname => localhost, port => 389, version => 3, timeout => 120, tls => 0, base => ou=people,dc=example,dc=com, scope => sub, query_filter => (&(objectclass=amavisaccount)(mail=%m)), bind_dn => cn=amavis,ou=services,dc=example,dc=com, bind_password => secret, };

62 amavisd-new LDAP Schema I

63 amavisd-new LDAP Schema II LDAP User Object using amavisaccount dn: objectclass: organizationalperson objectclass: person objectclass: top objectclass: PostfixBookMailAccount objectclass: extensibleobject objectclass: amavisaccount amavisbannedadmin: amavislocal: TRUE cn: Patrick Koetter givenname: Koetter mail: mailalias: mailalias: mailenabled: TRUE mailgidnumber: 2000 mailhomedirectory: /srv/mail/example.com/patrick mailquota: mailstoragedirectory: maildir:/srv/mail/example.com/patrick/maildir mailuidnumber: 2000 sn: Patrick uniqueidentifier: userpassword:: ye6xu9th==

64 Why use Policy Banks? Run one instance of amavisd-new Offer different behaviours (policies) Use port and/or client IP to identify policy bank

65 How Policy Banks work

66 An Example Policy Bank I A policy bank ALT $policy_bank{ ALT } = { log_level => 3, syslog_ident => alt-amavis, syslog_facility => LOCAL3, inet_acl => [qw( )], final_spam_destiny => D_PASS, final_bad_header_destiny => D_PASS, forward_method => smtp:*:*, notify_method => smtp:[ ]:10025, virus_admin_maps => "abuse\@alt\.example\.com", spam_lovers_maps => [qw( abuse@example.com )] ],... };

67 An Example Policy Bank II Assign policy to port $interface_policy{ } = ALT ;

68 Beyond the standard stuff

69 Keeping SpamAssassin up to date Spammers use SpamAssassin to optimize their spam Patterns start to get old the day you download them Use sa-update to update your SpamAssassin rules sa-update creates a new directory SpamAssassin first loads the packaged rules and then the updated ones

70 Extending SpamAssassins rules The standard ruleset has a narrow score corridor Rules Emporium adds more weight to scores Rules Emporium adds spezialized rules Use sa-update to install and update these rules

71 Extending ClamAV patterns ClamAV is just a content inspection engine Additional rule patterns add additional functionality Use for Phishing and Scam Signatures...

72 Pen-Pals Question Is it spam if you start talking about sex and your friend replies? amavisd-new can track who started a discussion apply lower scores to replies in a discussion that had been started by you

73 p0f - passive operating system fingerprinting p0f is a passive OS fingerprinting tool identifies the OS (type, version, patch level, etc.) on machines that connect to your box (SYN mode)

74 A p0f fingerprint in a message

75 p0f-analyzer.pl amavisd-new queries p0f via the p0f-analyzer.pl script passes the OS information as a header field to SpamAssassin SpamAssassin rules apply scores depending on the OS detected Invocation of p0f # Query p0f-analyzer.pl $os_fingerprint_method = p0f:*:2345 ;

76 Some statistics collected from Marc s logs in February 2006: p0f OS guess ham : spam Windows-XP 0.7 % : 99.3 % Windows % : 94.2 % UNKNOWN 16.5 % : 83.5 % Linux 58.8 % : 41.2 % Unix 80.3 % : 19.7 % (Unix+Linux 66.5 % : 33.5 %) ham mail with score below 3 spam score above 6

77 Scores in SA s local.cf describe L_P0F_EXISTS A header field X-Amavis-OS-Fingerprint does exist header L_P0F_EXISTS exists:x-amavis-os-fingerprint describe L_P0F_WXP Remote system is truly a Windows XP, not Windows 2000 header L_P0F_WXP X-Amavis-OS-Fingerprint =~ /\AWindows XP(?![^(]*\b2000 SP)/m score L_P0F_WXP 2.3 describe L_P0F_W Remote system is some Windows variant, except Win. XP header L_P0F_W X-Amavis-OS-Fingerprint =~ /\AWindows(?! XP)/m score L_P0F_W 1.3 describe L_P0F_UNKN P0f was unable to determine remote OS type header L_P0F_UNKN X-Amavis-OS-Fingerprint =~ /\AUNKNOWN/m score L_P0F_UNKN 0.8 describe L_P0F_Unix Remote system is running Unix, not Linux header L_P0F_Unix X-Amavis-OS-Fingerprint =~ /\A((Free Open Net)BSD Solaris \ HP-UX Tru64 AIX)/m score L_P0F_Unix -1.0 describe L_P0F_Unix Remote system is running Linux header L_P0F_Linux X-Amavis-OS-Fingerprint =~ /\ALinux/m score L_P0F_Linux -0.1

78 DKIM - Domainkeys Identified Mail Goal Sign messages using a digital identity Establish a good sender reputation Use sender reputation to tell good from neutral or bad messages DKIM is a standard (RFC 4871). It describes a framework to cryptographically sign messages cryptographically verify messages

79 How DKIM works...

80 How to apply DKIM signatures in amavisd-new... Version You need amavisd-new 2.6 or later to implement DKIM Cryptographic keys Create a pair of cryptographic keys: # amavisd genrsa /path/to/keylocation/selector DKIM configuration example $enable_dkim_verification = 1; $enable_dkim_signing = 1; # signing domain selector private key options dkim_key ( example.org, abc, /var/db/dkim/a.key.pem );

81 How DKIM works...

82 How to verify DKIM signatures in SpamAssassin... Loading the plugin in v320.pre loadplugin Mail::SpamAssassin::Plugin::DKIM Setting DKIM rules in local.cf score DKIM_VERIFIED -1.3 score DKIM_POLICY_TESTING 0 score USER_IN_DKIM_WHITELIST -4.0 whitelist_from_dkim *@state-of-mind.de state-of-mind.de whitelist_from_dkim *@intl.paypal.com paypal.com

83 Questions? Ralf Hildebrandt Patrick Ben Koetter Slides amavisd.pdf