Solving Large-Scale Database Administration with Tungsten

Size: px

Start display at page:

Download "Solving Large-Scale Database Administration with Tungsten"

Randolph McKenzie
10 years ago
Views:

1 Solving Large-Scale Database Administration with Tungsten Neil Armitage, Sr. Software Engineer Robert Hodges, CEO.

2 Introducing Continuent The leading provider of clustering and replication for open source DBMS Our Product: Continuent Tungsten Clustering - Commercial-grade HA, performance scaling and data management for MySQL Replication - Flexible, high-performance data movement 2

Clustering - Commercial-grade HA, performance scaling and data

3 Quick Continuent Facts Largest Tungsten clustering installation processes about 500M transactions each day Tungsten Replicator was application of the year at the 2011 MySQL User Conference Continuent is well known in the MySQL community for its clustering and replication Over 90 customers currently deployed 3

year at the 2011 MySQL User Conference Continuent is well known in the MySQL

4 Select Continuent Customers 4

5 Why Tungsten? 5

6 MySQL is a Great OLTP Database But how do I... Handle DBMS failures? Perform maintenance and upgrades without taking applications o!ine? Load balance SQL across replicas? Create a disaster recovery site? Set up multi-master replication? Integrate with another DBMS? 6

Load balance SQL across replicas? Create a disaster recovery site?

7 Tungsten Master/Slave Clustering 7

8 Introducing Tungsten Clustering Tungsten clusters combine o"- the-shelf open source DBMS servers into data services with: 24x7 data access Scaling of load on replicas Simple management commands Connector GonzoPortal.com apache /php Connector...without app changes or data migration Amazon US West 8

of load on replicas Simple management commands Connector GonzoPortal.

9 Doesn t Replication Do This Already? Application Libmysqlclient.a Application Libmysqlclient.a Db2 Db1 Db3 Replicator Slave Replicator Master Replicator Slave 9

10 Replication Limitations Replication alone cannot enable constant access to data... What happens if the master fails? What happens if a slave is lagging too far behind to be usable? How do I maintain the DBMS without stopping apps? 10

What happens if a slave is lagging too far behind to be

11 From Replicating to Clustering Application Application Tungsten Connector Tungsten Connector Monitoring and Control Db2 Db1 Db3 Monitoring and Control Manager Manager Manager Replicator Slave Replicator Master Data Service: sjc Replicator Slave 11

Db2 Db1 Db3 Monitoring and Control Manager Manager Manager

12 Basic Cluster Operations Switch -- Promote a slave node to master Shun -- Kick node out of cluster Welcome -- Return node to cluster Backup -- Run a backup Restore -- [Re-]provision a node Recover -- Return a broken node to slave Heartbeat -- Check cluster liveness 12

Backup -- Run a backup Restore -- [Re-]provision a node Recover --

13 Automatic Failover Application Tungsten Connector 13

14 Automatic Failover Application Tungsten Connector 2. Halt master connections 1. Detect nonresponsive master 14

15 Automatic Failover Application Tungsten Connector 3. Select most advanced slave and wait for pending transactions to apply 15

16 Automatic Failover Application Tungsten Connector 4. Promote to master, reconfigure slave(s), and re-enable connections 16

17 Automatic Failover Application Tungsten Connector 5. Recover failed master 17

18 Rolling SQL Schema Upgrade Slave Upgrade Slave Upgrade Switch Final Upgrade Shun slave 1, run upgrade, and return to cluster. Discard and recover on failure Repeat for additional slave(s) Switch master to promote an upgraded slave Upgrade old master. Cluster schema is now fully upgraded 18

Discard and recover on failure Repeat for additional slave(s) Switch

19 Backup, Recovery, and Restore Backups /cluster > datasource DB1 backup Recover /cluster > datasource DB2 recover Restore /cluster > datasource DB2 restore Out-of-box support for mysqldump/ XtraBackup; simple integration for others 19

recover Restore /cluster > datasource DB2 restore

20 Understanding Tungsten Connectivity 20

21 Tungsten Connector Basics Any MySQL/PostgreSQL client can connect Initiates connections using JDBC App Native protocol COM_QUERY COM_INIT_DB COM_DROP_DB Connector 21

22 Tungsten Connector Basics Pass-through operation after connection Full transparency & low overhead for clients Protocol packet COM_QUERY select * from t App Connector Protocol packet OK ResultSet Rows:1 22

23 Simple Read/Write Splitting Very fast, easy-to-understand scaling model Transactions go to master Auto-commit SELECTs go to slaves, if available, but... Reads potentially outdated data Ephemeral SQL objects won t be available May miss: select update_function() 23

24 SmartScale Session Load Balancing Master DBMS Transaction History Log App Connect/Write Connector Session x Session y Slave replay position Slave DBMS 24

25 SmartScale Session Load Balancing Master DBMS Transaction History Log App Read Connector Session x Session y Slave replay position Slave DBMS 25

26 SmartScale Session Load Balancing Master DBMS Transaction History Log App Read Connector Session x Slave replay position Slave DBMS 26

27 Tungsten DBMS Array Connectivity cluster1 login App Read/Write Connector cluster2 login cluster3 login 27

28 Deployment on App Server Hosts App Server + Connector App Server + Connector App Server + Connector App Server + Connector App Server + Connector 28

29 Deployment on Dedicated Hosts App Server App Server App Server App Server App Server Hardware Load Balancer 29

30 Deployment on Dedicated Hosts App Server App Server App Server App Server App Server Software Load Balancer 30

31 Managing Data Across Multiple Sites 31

32 Disaster Recovery Sites Updates go to primary site master Simple command promotion DR site receives updates in real time Tungsten connectors can route to primary and DR sites No application changes required Simplest way to protect against region/site failures 32

33 Tungsten Composite Data Services App Server + Connector App Server + Connector San Jose New York Master sj_nyc Slave 33

34 Switching and failover Planned switch /sj_nyc > switch to nyc Failover /cluster > failover Failover between sites is not automated! 34

35 Multi-Master Replication Updates on 2+ sites (active-active mode) Enables geographic distribution of data No failover necessary if network fails or site becomes unavailable Not all applications can handle multi-master Applications must avoid con#icts Careful testing required Restoration of broken systems may not be easy 35

36 Simple Multi-Master Con$guration NYC Replicator Replicator Frankfurt fra (slave) fra (master) nyc (master) nyc (slave) 36

37 Complex Multi-Master Topologies All Masters Star Replication 37

38 Fan-In Replication Multiple masters feed into a single slave Use case: Aggregate data for reporting Consolidate data from multiple servers into one server Moving shards Applications must avoid con#icts 38

39 Fan-In Topology 39

40 Heterogeneous Replication 40

41 Problem: Real-Time Order Ful$llment Web-Facing Sales Purchase Ordering PHP/ Perl Java/ JBoss MySQL 1-2M transactions daily Real-Time Replication Oracle 3M Order Items 41

42 How Does Tungsten Do That? MySQL Tungsten Master Replicator Tungsten Slave Replicator Oracle Service oracle Service oracle MySQL Binlog MySQLExtractor Special Filters * Transform enum to string Special Filters * Ignore extra tables * Map names to upper case * Optimize updates to remove unchanged columns binlog_format=row 42

43 Batch Loading to Data Warehouses MySQL Master Replicator Service batch Vertica Data Warehouse London Transaction History Log THL Csv File Csv File... Csv File 43

44 Possibilities for Analytics... MySQL Master Complex, near real-time reporting OLTP Data Light-weight, real-time operational status Web-facing minidata marts for SaaS users 44

45 Getting Started 45

46 How Do I Obtain Tungsten? Register at Continuent to get downloads for Oracle replication: Documentation is available on Continuent website MySQL replication is available from Tungsten Replicator project on code.google.com 46

47 560 S. Winchester Blvd., Suite 500 San Jose, CA Tel +1 (866) Fax +1 (408) Our Blogs: Continuent Web Page: Tungsten Replicator 2.0:

How, What, and Where of Data Warehouses for MySQL

How, What, and Where of Data Warehouses for MySQL Robert Hodges CEO, Continuent. Introducing Continuent The leading provider of clustering and replication for open source DBMS Our Product: Continuent Tungsten