Timo Sirainen Dovecot Oy
Talk Overview Quick introduc?on Tips & quick feature overviews dsync- based replica?on Ques?ons
Dovecot? IMAP, POP3 mail server Mail delivery agent + LMTP Sieve mail filtering language + ManageSieve (by Stephan Bosch) (No SMTP. PosSix/Exim is fine) Now, an SMTP submission server maybe..
Dovecot Features High performance (low disk I/O usage) Highly configurable/flexible Modular code, plugins can do ~anything Admin- friendly All errors are logged Understandable error messages Automated fixing of (corrup?on) errors
PosSix Integra?on SMTP AUTH via Dovecot auth Mail delivery via LMTP User lookups via tcp service to Dovecot auth RCPT TO- stage over quota reject via Dovecot quota- status policy server Only aliases need to be configured to PosSix
Logging Error log should stay empty! Separate error log makes this easy to check. All errors are either due to misconfigura?on, server trouble or Dovecot bug..
Mail Dele?on Logging For answering Why did you lose my emails? h2p://wiki2.dovecot.org/plugins/maillog imap(user): copy - > Trash: uid=908, msgid=<123.foo@bar> imap(user): deleted: uid=908, msgid=<123.foo@bar> imap(user): expunged: uid=908, msgid=<123.foo@bar>
Sessions in Log Each IMAP/POP3 connec?on gets session ID Each log line can include session ID imap- login, pop3- login auth imap, pop3 Session ID passes through Dovecot proxies mail_log_prefix = %s(%u,<%{session}>):
Why Is It Slow? CPU usage? Memory (swap) usage? Disk I/O? SSD to indexes stats & imap_stats plugin h2p://www.dovecot.org/tools/stats- top.pl h2p://wiki2.dovecot.org/performancetuning
Mailbox Formats Maildir for super- reliability mdbox / sdbox for performance Tries very hard to recover from corrup?on Single instance a2achment storage For mdbox & sdbox formats
Full Text Search Indexes Lucene Easy, no server Somewhat old.. Solr Java server Up- to- date Lucene (Squat is deprecated) Easy to add more!
Mailboxes Ideally same mailbox names for everyone Sent, Trash, Draps, Junk Auto- create? Auto- subscribe? IMAP SPECIAL- USE becoming more common Client translates to local language Gmail, ios, Outlook 2013 See example- config/conf.d/15- mailboxes.conf
Virtual Mailboxes All messages in all mailboxes Search from all mails Unseen messages in all mailboxes Gmail- like conversa?on views Very useful for customized webmails
Lazy Expunge Allow users to undelete mails Move mails to another namespace without quota Delay final expunging un?l later e.g. nightly cronjob dele?ng 1w old mails or combine with quota+trash plugins to keep lazy- expunge quota Virtual all expunged mails
doveadm mail commands fetch expunge copy move
doveadm import Proper way to import mails from a backup Or copy mails between users Search query to select mails
doveadm exec doveadm exec imap u user@domain No password required For quick tes?ng For scrip?ng: prins a select inbox\n b search return (save) from user1@domain\n c store $ +flags \\seen\n d logout\n doveadm exec imap u user@domain
doveadm dump See contents of Dovecot s binary files indexes Lucene FTS indexes IMAP COMPRESS DEFLATE stream For debugging mainly
dsync- Based Migra?on doveadm sync / doveadm backup Between any IMAP/POP3 <- > Dovecot Also Dovecot <- > Dovecot! (mailbox format, filesystem, server, etc. change) Zero (meta)data loss Zero down?me h2p://wiki2.dovecot.org/migra?on/dsync
Replica?on Possibili?es Block device (e.g. DRBD) FS corrup?on replicated async=data loss, sync=too slow unless near Cluster POSIX filesystems Open slow, complex, buggy Object storage Simpler cluster filesystems = easier to make it work! Open high latency Dovecot object storage plugin Applica?on- level replica?on (dsync)
dsync- Based Replica?on dsync does two- way merging of mailboxes Both sides can do changes - > no data loss Replica?on triggers run dsync asynchronously Over SSH / TCP / TCP+SSL Geographically distributed cluster Two virtual servers from different companies - > cheap HA mail cluster
Incoming Email Incoming Email IMAP User Read replicated mail Dovecot -Proxy/Director -Backend -SMTP Server Datacenter Helsinki dsync replication over SSH/TCP/TCP+SSL Dovecot - Proxy/Director - Backend - SMTP Server Datacenter Tokyo
dsync- Replica?on Efficiency v2.2 supports stateful sync replicator does full sync if no state full sync periodically (just in case) stateful sync for each change a few kb of traffic Saving new mails have priority
dsync- Replica?on Issues IMAP UID renumbering for conflic?ng new mails IMAP clients redownload mails Some other (mainly theore?cal) conflicts Avoid conflicts by accessing user s mails always via the same backend Unless it s down
dsync- Replica?on Load Balancing Load balancing / HA by: a) Actual load balancer b) Mul?ple IPs in DNS A records (low TTL) Dovecot proxy/director answers and redirects to backend If the primary backend is down, use the other With widely separated geo- clusters proxying adds latency.. No proxying, but geo- DNS / rou?ng?
Ques?ons?