Timo Sirainen Dovecot Oy h2p://

Transcription

1 Timo Sirainen Dovecot Oy

2 Talk Overview Quick introduc?on Tips & quick feature overviews dsync- based replica?on Ques?ons

3 Dovecot? IMAP, POP3 mail server Mail delivery agent + LMTP Sieve mail filtering language + ManageSieve (by Stephan Bosch) (No SMTP. PosSix/Exim is fine) Now, an SMTP submission server maybe..

4 Dovecot Features High performance (low disk I/O usage) Highly configurable/flexible Modular code, plugins can do ~anything Admin- friendly All errors are logged Understandable error messages Automated fixing of (corrup?on) errors

5 PosSix Integra?on SMTP AUTH via Dovecot auth Mail delivery via LMTP User lookups via tcp service to Dovecot auth RCPT TO- stage over quota reject via Dovecot quota- status policy server Only aliases need to be configured to PosSix

6 Logging Error log should stay empty! Separate error log makes this easy to check. All errors are either due to misconfigura?on, server trouble or Dovecot bug..

7 Mail Dele?on Logging For answering Why did you lose my s? h2p://wiki2.dovecot.org/plugins/maillog imap(user): copy - > Trash: uid=908, msgid=<123.foo@bar> imap(user): deleted: uid=908, msgid=<123.foo@bar> imap(user): expunged: uid=908, msgid=<123.foo@bar>

8 Sessions in Log Each IMAP/POP3 connec?on gets session ID Each log line can include session ID imap- login, pop3- login auth imap, pop3 Session ID passes through Dovecot proxies mail_log_prefix = %s(%u,<%{session}>):

9 Why Is It Slow? CPU usage? Memory (swap) usage? Disk I/O? SSD to indexes stats & imap_stats plugin h2p:// top.pl h2p://wiki2.dovecot.org/performancetuning

10 Mailbox Formats Maildir for super- reliability mdbox / sdbox for performance Tries very hard to recover from corrup?on Single instance a2achment storage For mdbox & sdbox formats

11 Full Text Search Indexes Lucene Easy, no server Somewhat old.. Solr Java server Up- to- date Lucene (Squat is deprecated) Easy to add more!

12 Mailboxes Ideally same mailbox names for everyone Sent, Trash, Draps, Junk Auto- create? Auto- subscribe? IMAP SPECIAL- USE becoming more common Client translates to local language Gmail, ios, Outlook 2013 See example- config/conf.d/15- mailboxes.conf

13 Virtual Mailboxes All messages in all mailboxes Search from all mails Unseen messages in all mailboxes Gmail- like conversa?on views Very useful for customized webmails

14 Lazy Expunge Allow users to undelete mails Move mails to another namespace without quota Delay final expunging un?l later e.g. nightly cronjob dele?ng 1w old mails or combine with quota+trash plugins to keep lazy- expunge quota Virtual all expunged mails

15 doveadm mail commands fetch expunge copy move

16 doveadm import Proper way to import mails from a backup Or copy mails between users Search query to select mails

17 doveadm exec doveadm exec imap u user@domain No password required For quick tes?ng For scrip?ng: prins a select inbox\n b search return (save) from user1@domain\n c store $ +flags \\seen\n d logout\n doveadm exec imap u user@domain

18 doveadm dump See contents of Dovecot s binary files indexes Lucene FTS indexes IMAP COMPRESS DEFLATE stream For debugging mainly

19 dsync- Based Migra?on doveadm sync / doveadm backup Between any IMAP/POP3 <- > Dovecot Also Dovecot <- > Dovecot! (mailbox format, filesystem, server, etc. change) Zero (meta)data loss Zero down?me h2p://wiki2.dovecot.org/migra?on/dsync

20 Replica?on Possibili?es Block device (e.g. DRBD) FS corrup?on replicated async=data loss, sync=too slow unless near Cluster POSIX filesystems Open slow, complex, buggy Object storage Simpler cluster filesystems = easier to make it work! Open high latency Dovecot object storage plugin Applica?on- level replica?on (dsync)

21 dsync- Based Replica?on dsync does two- way merging of mailboxes Both sides can do changes - > no data loss Replica?on triggers run dsync asynchronously Over SSH / TCP / TCP+SSL Geographically distributed cluster Two virtual servers from different companies - > cheap HA mail cluster

22 Incoming Incoming IMAP User Read replicated mail Dovecot -Proxy/Director -Backend -SMTP Server Datacenter Helsinki dsync replication over SSH/TCP/TCP+SSL Dovecot - Proxy/Director - Backend - SMTP Server Datacenter Tokyo

23 dsync- Replica?on Efficiency v2.2 supports stateful sync replicator does full sync if no state full sync periodically (just in case) stateful sync for each change a few kb of traffic Saving new mails have priority

24

25 dsync- Replica?on Issues IMAP UID renumbering for conflic?ng new mails IMAP clients redownload mails Some other (mainly theore?cal) conflicts Avoid conflicts by accessing user s mails always via the same backend Unless it s down

26 dsync- Replica?on Load Balancing Load balancing / HA by: a) Actual load balancer b) Mul?ple IPs in DNS A records (low TTL) Dovecot proxy/director answers and redirects to backend If the primary backend is down, use the other With widely separated geo- clusters proxying adds latency.. No proxying, but geo- DNS / rou?ng?

27 Ques?ons?