Parallel Replication for MySQL in 5 Minutes or Less

Parallel Replication for MySQL in 5 Minutes or Less Featuring Tungsten Replicator Robert Hodges, CEO, Continuent

About Continuent / Continuent is the leading provider of data replication and clustering for open source relational databases / Our Products: Tungsten Replicator - High-performance, MySQL replication Tungsten Enterprise - Commercial replication and data management solution for MySQL and PostgreSQL / Our Services: Consulting on Tungsten plus replication and clustering in general Subscriptions for commercial products

Adding Parallel Replication in 5 Minutes or Less

Problem: Single Threaded Replication Master Slave IO and SQL threads I/O Thread downloads binlog I/O thread writes relay logs; updates coords in master.info SQL Thread reads and applies relay logs; updates coords in relay- logs.info master.info Relay logs relay-log.info MySQL SQL thread is overwhelmed by CPU- or IO-bound queries

Solution: Tungsten Parallel Slave Master Slave IO and SQL threads disabled Client connections Tungsten downloads binlog Tungsten Replicator Tungsten converts binlogs to local log, then applies to slave using parallel connections Disk logs

Installing Tungsten Parallel Replication $ tar -xvzf tungsten-replicator-2.0.5-357.tar.gz $ cd tungsten-replicator-2.0.5-357 $ tools/tungsten-installer tools/tungsten-installer --direct -a \ --service-name=parallel --native-slave-takeover \ --master-host=127.0.0.1 --master-port=33306 \ --master-user=msandbox --master-password=msandbox \ --slave-host=127.0.0.1 --slave-port=33307 \ --slave-user=msandbox --slave-password=msandbox \ --home-directory=/opt/continuent \ --property=replicator.store.parallel-queue.maxofflineinterval=5 \ --svc-parallelization-type=disk --buffer-size=100 \ --channels=30 --thl-port=2115 --rmi-port=10010 \ --skip-validation-check=mysqlpermissionscheck \ --skip-validation-check=mysqlapplierserveridcheck \ --start-and-report (Red options required for MySQL sandboxes)

Understanding Tungsten Parallel Replication

What Is Tungsten Replicator? Tungsten Replicator is a fast, opensource replication engine for open source databases GPL V2 license Written in Java Designed for speed and flexibility http://code.google.com/p/tungsten-replicator

Tungsten Replicator Architecture Master (Replicator configuration) MySQL Tail binlog or login as client Tungsten Replicator (replicator) replicator. properties Binlogs Transaction History Log Transport via TCP/IP connection (Transactions + Metadata) Transaction History Log replicator. properties Tungsten Replicator (replicator) Slave Apply using JDBC MySQL

How Can We Parallelize? / Look for workloads that have independent streams of updates Shared data requires full serialization shared tenant_1 tenant_2 tenant_3 tenant_4 Tenants independent from each other but depend on shared data shared tenant_1 tenant_2 tenant_3 tenant_4 Tenant database updates can move in parallel streams

Replicator Pipelines and Parallel Apply Tungsten Replicator Process Pipeline Extract Stage Assign Shard ID Apply Extract Stage Apply Parallel Queue Extract Extract Extract Stage Apply Apply Apply channels Transaction History Log Binlog THL shard.list file Slave DBMS

Sharding Rules for Safe Parallel Apply SQL Statement/Row Update use myschema; create table foo (id int); use myschema; create table yourschema.foo (id int); begin; insert into yourschema.foo values(1); commit; begin; insert into myschema.foo values(1); insert into yourschema.foo values(1); commit; Shard ID myschema #UNKNOWN (full serialization req d) yourschema #UNKNOWN (full serializationreq d) Summary: Serialize if parallel apply is unsafe

Parallel On-Disk Queue Global sync counter (Store) THL (Stage) thl-to-q THLParallel Queue 1 2 3 (Stage) q-to-dbms channels DBMS Per-channel read threads Queue per channel

How Fast Is Tungsten?

Understanding What to Test Application Application Application Typical web property: 1.) Data size >> buffer pool 2.) Storage typically on disk 3.) Heavy read traffic + writes 4.) Slaves lag after maintenance Master InnoDB Buffer Pool Slave InnoDB Buffer Pool Disk/SSD Storage Disk/SSD Storage

Sysbench Performance Tests Head-to-head Tungsten vs. MySQL replication Test Scenario Databases Rows/Db Data Size Cache-resident 30 10K 430Mb I/O Bound 30 10M 68Gb HP Proliant Server w Dual Xeon L5520, 72 GB RAM, 1TB HP Smart Array RAID 1+0 MySQL Version 5.1.57, 10Gb InnoDB buffer pool Run 1 hour sysbench OLTP load Start slave with empty buffer pool Measure throughput and total catch-up

Cache-Resident Total Binlog Comparison Tungsten Replicator (~17 minutes) MySQL Replication (~30 minutes)

Cache-Resident Throughput Comparison Tungsten ~1.8x faster than MySQL; both exceed master Tungsten Replicator (~17 minutes) MySQL Replication (~30 minutes)

I/O Bound Total Binlog Comparison Tungsten Replicator (~51 minutes) MySQL Replication (~228 minutes)

I/O Bound Throughput Comparison Tungsten ~4.5x faster than MySQL; Only Tungsten exceeds master throughput Tungsten Replicator (~51 minutes) MySQL Replication (~228 minutes)

Tips for Maximizing Parallel Replication Performance

Pick the Right Workload / Parallel replication is great for I/O bound workloads Small buffer pool compared to dataset size Very large data sets Slaves that also take read-only queries / Cache-resident workloads see less benefit Dataset < buffer cache size Few I/O bound updates

Pick the Right Application Profile / Multi-tenant applications Independent customer databases Minimal or no shared data between tenants Uniform distribution of updates across schemas / Horizontally shared applications Data distributed across schemas No cross-shard queries (Rule of thumb: >1-2% serialization kills parallel apply performance)

Allocate Capable Hardware / Ensure enough RAM Large InnoDB buffer pool 1GB RAM for Tungsten Replicator JVM 500M-2GB OS page cache for parallel replication / Fast storage BBU for fast fsync RAID Good controller cache to buffer updates / 1-2 CPUs required for replication / Run off-board to avoid master impact (Not enough testing on SSD to offer recommendations yet)

Tune Tungsten Properly In-memory index Sequential writes, random reads Sequential writes only (best case) Java Virtual Machine 1 writer thread, 0+ reader threads Java Stream Classes OS Page Cache Disk/SSD Log options buffersize - 128Kb dochecksum - Costs CPU but probably worth it fsynconflush - Slow on non-bbu storage but required for crash-save slaves logfilesize - If bigger seek time on start-up is longer OS Settings Ensure at least 1GB in page cache Use Innodb O_DIRECT if onboard with MySQL Storage settings Separate device best but not required

Where Parallel Apply Does Not Help / Single large database Difficult to parallelize transactions safely Tungsten serializes everything / Cross-schema queries Multi-tenant app with shared data between tenants Messy single-application spread across schemas Tungsten serializes cross-schema updates / Economy-class hardware or VMs Adding more threads does not help if one thread already hogs the disk

Off On Your Own

Home Sweet Home http://code.google.com/p/tungsten-replicator

More Than Just Parallel Apply / Global transaction IDs / Flexible transaction filters / Replicate from MySQL to PG/Oracle/MongoDB / Backup and restore integration / Cross-version replication: 5.5 -> 5.1 -> 5.0 -> 4.1 / Row and statement replication / Automatic consistency checks / Multi-master replication / Parallel replication (of course!) / And extensible too

More Tungsten Talks / 1:30pm MySQL Parallel Replication in 5 minutes or less (Robert Hodges) / 2:30pm MySQL Replication outside the Box: Multiple Masters, Fan-in, Parallel Apply (Giuseppe Maxia) / 4:30pm MySQL Sandbox: a Framework for Productive Laziness (Giuseppe Maxia)

Conclusion / Tungsten parallel replication works for MySQL version 5.0+ / You can enable slave takeover in a few minutes / Tungsten Replicator is up to 4.5x faster than MySQL built-in replication Try out Tungsten Replicator today!

Contact Information for Continuent HQ 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009 e-mail: sales@continuent.com Our blogs http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://flyingclusters.blogspot.com Continuent Web Site: http://www.continuent.com Tungsten Replicator 2.0: http://code.google.com/p/tungsten-replicator