Parallel Replication for MySQL in 5 Minutes or Less

Size: px

Start display at page:

Download "Parallel Replication for MySQL in 5 Minutes or Less"

Jonah Wilcox
10 years ago
Views:

1 Parallel Replication for MySQL in 5 Minutes or Less Featuring Tungsten Replicator Robert Hodges, CEO, Continuent

2 About Continuent / Continuent is the leading provider of data replication and clustering for open source relational databases / Our Products: Tungsten Replicator - High-performance, MySQL replication Tungsten Enterprise - Commercial replication and data management solution for MySQL and PostgreSQL / Our Services: Consulting on Tungsten plus replication and clustering in general Subscriptions for commercial products

Enterprise - Commercial replication and data management solution for MySQL and PostgreSQL / Our

3 Adding Parallel Replication in 5 Minutes or Less

4 Problem: Single Threaded Replication Master Slave IO and SQL threads I/O Thread downloads binlog I/O thread writes relay logs; updates coords in master.info SQL Thread reads and applies relay logs; updates coords in relay- logs.info master.info Relay logs relay-log.info MySQL SQL thread is overwhelmed by CPU- or IO-bound queries

info SQL Thread reads and applies relay logs; updates coords in relay- logs.

5 Solution: Tungsten Parallel Slave Master Slave IO and SQL threads disabled Client connections Tungsten downloads binlog Tungsten Replicator Tungsten converts binlogs to local log, then applies to slave using parallel connections Disk logs

binlog Tungsten Replicator Tungsten converts binlogs to

6 Installing Tungsten Parallel Replication $ tar -xvzf tungsten-replicator tar.gz $ cd tungsten-replicator $ tools/tungsten-installer tools/tungsten-installer --direct -a \ --service-name=parallel --native-slave-takeover \ --master-host= master-port=33306 \ --master-user=msandbox --master-password=msandbox \ --slave-host= slave-port=33307 \ --slave-user=msandbox --slave-password=msandbox \ --home-directory=/opt/continuent \ --property=replicator.store.parallel-queue.maxofflineinterval=5 \ --svc-parallelization-type=disk --buffer-size=100 \ --channels=30 --thl-port= rmi-port=10010 \ --skip-validation-check=mysqlpermissionscheck \ --skip-validation-check=mysqlapplierserveridcheck \ --start-and-report (Red options required for MySQL sandboxes)

$0.0.1 --slave-port=33307 \ --slave-user=msandbox --slave-password=msandbox \ --home-directory=/opt/continuent \ --property=replicator.store.parallel-queue.$

7 Understanding Tungsten Parallel Replication

8 What Is Tungsten Replicator? Tungsten Replicator is a fast, opensource replication engine for open source databases GPL V2 license Written in Java Designed for speed and flexibility

9 Tungsten Replicator Architecture Master (Replicator configuration) MySQL Tail binlog or login as client Tungsten Replicator (replicator) replicator. properties Binlogs Transaction History Log Transport via TCP/IP connection (Transactions + Metadata) Transaction History Log replicator. properties Tungsten Replicator (replicator) Slave Apply using JDBC MySQL

properties Binlogs Transaction History Log Transport via TCP/IP connection (Transactions

10 How Can We Parallelize? / Look for workloads that have independent streams of updates Shared data requires full serialization shared tenant_1 tenant_2 tenant_3 tenant_4 Tenants independent from each other but depend on shared data shared tenant_1 tenant_2 tenant_3 tenant_4 Tenant database updates can move in parallel streams

requires full serialization shared tenant_1 tenant_2 tenant_3 tenant_4 Tenants

11 Replicator Pipelines and Parallel Apply Tungsten Replicator Process Pipeline Extract Stage Assign Shard ID Apply Extract Stage Apply Parallel Queue Extract Extract Extract Stage Apply Apply Apply channels Transaction History Log Binlog THL shard.list file Slave DBMS

Apply Parallel Queue Extract Extract Extract Stage Apply Apply

12 Sharding Rules for Safe Parallel Apply SQL Statement/Row Update use myschema; create table foo (id int); use myschema; create table yourschema.foo (id int); begin; insert into yourschema.foo values(1); commit; begin; insert into myschema.foo values(1); insert into yourschema.foo values(1); commit; Shard ID myschema #UNKNOWN (full serialization req d) yourschema #UNKNOWN (full serializationreq d) Summary: Serialize if parallel apply is unsafe

foo values(1); commit; begin; insert into myschema.foo values(1); insert into yourschema.

13 Parallel On-Disk Queue Global sync counter (Store) THL (Stage) thl-to-q THLParallel Queue (Stage) q-to-dbms channels DBMS Per-channel read threads Queue per channel

14 How Fast Is Tungsten?

15 Understanding What to Test Application Application Application Typical web property: 1.) Data size >> buffer pool 2.) Storage typically on disk 3.) Heavy read traffic + writes 4.) Slaves lag after maintenance Master InnoDB Buffer Pool Slave InnoDB Buffer Pool Disk/SSD Storage Disk/SSD Storage

) Storage typically on disk 3.) Heavy read traffic + writes 4.

16 Sysbench Performance Tests Head-to-head Tungsten vs. MySQL replication Test Scenario Databases Rows/Db Data Size Cache-resident 30 10K 430Mb I/O Bound 30 10M 68Gb HP Proliant Server w Dual Xeon L5520, 72 GB RAM, 1TB HP Smart Array RAID 1+0 MySQL Version , 10Gb InnoDB buffer pool Run 1 hour sysbench OLTP load Start slave with empty buffer pool Measure throughput and total catch-up

Bound 30 10M 68Gb HP Proliant Server w Dual Xeon L5520, 72 GB RAM, 1TB HP Smart Array RAID 1+0

17 Cache-Resident Total Binlog Comparison Tungsten Replicator (~17 minutes) MySQL Replication (~30 minutes)

18 Cache-Resident Throughput Comparison Tungsten ~1.8x faster than MySQL; both exceed master Tungsten Replicator (~17 minutes) MySQL Replication (~30 minutes)

19 I/O Bound Total Binlog Comparison Tungsten Replicator (~51 minutes) MySQL Replication (~228 minutes)

20 I/O Bound Throughput Comparison Tungsten ~4.5x faster than MySQL; Only Tungsten exceeds master throughput Tungsten Replicator (~51 minutes) MySQL Replication (~228 minutes)

21 Tips for Maximizing Parallel Replication Performance

22 Pick the Right Workload / Parallel replication is great for I/O bound workloads Small buffer pool compared to dataset size Very large data sets Slaves that also take read-only queries / Cache-resident workloads see less benefit Dataset < buffer cache size Few I/O bound updates

23 Pick the Right Application Profile / Multi-tenant applications Independent customer databases Minimal or no shared data between tenants Uniform distribution of updates across schemas / Horizontally shared applications Data distributed across schemas No cross-shard queries (Rule of thumb: >1-2% serialization kills parallel apply performance)

24 Allocate Capable Hardware / Ensure enough RAM Large InnoDB buffer pool 1GB RAM for Tungsten Replicator JVM 500M-2GB OS page cache for parallel replication / Fast storage BBU for fast fsync RAID Good controller cache to buffer updates / 1-2 CPUs required for replication / Run off-board to avoid master impact (Not enough testing on SSD to offer recommendations yet)

25 Tune Tungsten Properly In-memory index Sequential writes, random reads Sequential writes only (best case) Java Virtual Machine 1 writer thread, 0+ reader threads Java Stream Classes OS Page Cache Disk/SSD Log options buffersize - 128Kb dochecksum - Costs CPU but probably worth it fsynconflush - Slow on non-bbu storage but required for crash-save slaves logfilesize - If bigger seek time on start-up is longer OS Settings Ensure at least 1GB in page cache Use Innodb O_DIRECT if onboard with MySQL Storage settings Separate device best but not required

26 Where Parallel Apply Does Not Help / Single large database Difficult to parallelize transactions safely Tungsten serializes everything / Cross-schema queries Multi-tenant app with shared data between tenants Messy single-application spread across schemas Tungsten serializes cross-schema updates / Economy-class hardware or VMs Adding more threads does not help if one thread already hogs the disk

27 Off On Your Own

28 Home Sweet Home

29 More Than Just Parallel Apply / Global transaction IDs / Flexible transaction filters / Replicate from MySQL to PG/Oracle/MongoDB / Backup and restore integration / Cross-version replication: 5.5 -> 5.1 -> 5.0 -> 4.1 / Row and statement replication / Automatic consistency checks / Multi-master replication / Parallel replication (of course!) / And extensible too

30 More Tungsten Talks / 1:30pm MySQL Parallel Replication in 5 minutes or less (Robert Hodges) / 2:30pm MySQL Replication outside the Box: Multiple Masters, Fan-in, Parallel Apply (Giuseppe Maxia) / 4:30pm MySQL Sandbox: a Framework for Productive Laziness (Giuseppe Maxia)

31 Conclusion / Tungsten parallel replication works for MySQL version 5.0+ / You can enable slave takeover in a few minutes / Tungsten Replicator is up to 4.5x faster than MySQL built-in replication Try out Tungsten Replicator today!

32 Contact Information for Continuent HQ 560 S. Winchester Blvd., Suite 500 San Jose, CA Tel (866) Fax (408) Our blogs Continuent Web Site: Tungsten Replicator 2.0:

Preparing for the Big Oops! Disaster Recovery Sites for MySQL. Robert Hodges, CEO, Continuent MySQL Conference 2011

Preparing for the Big Oops! Disaster Recovery Sites for Robert Hodges, CEO, Continuent Conference 2011 Topics / Introductions / A Motivating Story / Master / Slave Disaster Recovery Replication Tungsten