Replicating to everything

Transcription

1 Replicating to everything Featuring Tungsten Replicator A Giuseppe Maxia, QA Architect Vmware

2 About me Giuseppe Maxia, a.k.a. "The Data Charmer" QA Architect at VMware Previously at AB / Sun / 3 times community award winner. 25+ years development and DB experience Creator and maintainer of -Sandbox ACE Director A Continuent

3 Presentation Topics Introductions Tungsten Replicator Overview Real-Time Data Warehouse Loading Real time replication to something else! Make your own replication Wrap-up 3

4 Introducing Continuent, a VMware company The leading provider of clustering and replication for open source DBMS Our Product: Continuent Tungsten Clustering - Commercial-grade HA, performance scaling and data management for Replication - Flexible, high-performance data movement 4

5 Continuent Tungsten Customers 1 Continuent

6 Quick Continuent Facts Largest Tungsten installation by data volume processes over 800 million transactions per day on 225 terabytes of relational data Largest installation by transaction volume handles up to 8 billion transactions daily Wide variety of topologies including,, Vertica, and Hadoop are in production now Acquired by VMware in

7 Quick Continuent Facts Largest Tungsten installation by data volume processes over 800 million transactions per day on 225 terabytes of relational data Largest the installation superheroes by transaction volume handles up to 8 billion transactions daily Wide variety of topologies including,, Vertica, and Hadoop are in production now We are of replication Acquired by VMware in

8 Tungsten Replicator Overview 7

9 What is Tungsten Replicator? A real-time, high-performance, open source database replication engine GPL V2 license - 100% open source Download from Annual support subscription available from Continuent Golden Gate without the Price Tag 8

10 Tungsten Replicator Overview Master Extract transactions from log Replicator THL (Transactions + Metadata) DBMS Logs Slave Replicator THL Apply (Transactions + Metadata) 9

11 master-slave fan-in slave all-masters Heterogeneous star

12 Heterogeneous One year ago...

13 Heterogeneous Now

14 Heterogeneous Now

15 Heterogeneous generic applier Now

16 Heterogeneous generic applier SQLite Now

17 Heterogeneous generic applier SQLite Now

18 Heterogeneous generic applier SQLite Now Whatever!

19 Real-Time Data Warehouse Loading 13

20 Current Data Warehouse Options Master Scalable row store with star schema and materialized view support CSV files stored on HDFS with access via Hive/MapReduce OLTP Data Column storage with compression and built-in time series support CSV files stored on S3 with live transformation 14

21 Traditional ETL Approach Batch-oriented = not timely Sales Extract Transfer Load Sales Table Table Scan for changes = performance hit Date columns = intrusive Data Warehouse 15

22 Real-Time Data Replication No SQL changes = transparent Data Warehouse Sales Table Data Replication Sales Table Fast propagation = timely DBMS Logs Automatic change capture = low impact 16

23 Loading to an Data Warehouse Tungsten Master Replicator Tungsten Slave Replicator oracle oracle Binlog Extractor Special Filters * Transform enum to string Special Filters * Ignore extra tables * Map names to upper case * Optimize updates to remove unchanged columns Data stored in rows like binlog_format=row 17

24 How about Loading to Hadoop? Provisioning Replication Binlog CSV CSV Files Buffered Files Transactions Data stored as append-only files 18

25 Typical Raw Hadoop Data Layout (CSV file) field separator record separator 556,MALTESE HOPE,4.99,127\n 557,MANCHURIAN CURTAIN,3.99,177\n 558,MANNEQUIN WORST,2.99,71\n 559,MARRIED GO,2.99,114\n file partitioning compression type conversions 19

26 Loading to a Hadoop Data Warehouse (Generate Hive table definitions) Transactions from master Javascript load script e.g. hadoop.js Replicator hadoop CSV CSV CSV Files Files Files Load using hadoop command Write data to CSV Staging Staging Tables Staging Tables Tables (Generate Hive table definitions) Base Tables Base Tables Materialized Views Merge using external MapReduce job 20

27 Loading to a Hadoop Data Warehouse (Generate Hive table definitions) Transactions from master Remember this! Javascript load script e.g. hadoop.js Replicator hadoop CSV CSV CSV Files Files Files Load using hadoop command Write data to CSV Staging Staging Tables Staging Tables Tables (Generate Hive table definitions) Base Tables Base Tables Materialized Views Merge using external MapReduce job 20

28 How about Loading to Vertica? Provisioning? Replication Binlog CSV CSV Files Buffered Files Transactions Data stored in column format 21

29 Loading to a Vertica Data Warehouse Tungsten Master Replicator Tungsten Slave Replicator vertica vertica Binlog binlog_format=row Extractor Special Filters * pkey - Fill in pkey info * colnames - Fill in names * replicate - Ignore tables CSV CSV Files CSV Files CSV Files CSV Files Files Large transaction batches to leverage load parallelization 22

30 Loading to a Vertica Data Warehouse Tungsten Master Replicator Tungsten Slave Replicator vertica vertica Binlog binlog_format=row Extractor Special Filters * pkey - Fill in pkey info * colnames - Fill in names * replicate - Ignore tables CSV CSV Files CSV Files CSV Files CSV Files Files Large transaction batches to leverage load parallelization Remember this! 22

31 Tungsten Support for Amazon Redshift Tungsten Replicator 3.0 supports real-time replication to Amazon Redshift Loading builds on data warehouse support for Vertica and Hadoop 23

32 Redshift = Cloud Data Warehouse 24

33 Tungsten Loading to Redshift Base Base Tables RedShift Tables Base Tables Transactions from Tungsten Master Tungsten Slave redshift Write data to CSV 3. Merge using SQL to base Staging Staging Tables Tables RedShift Staging Tables Javascript load script e.g. hadoop.js CSV CSV Files CSV Files Files 1. Load to S3 using s3cmd S3 Storage 2. COPY to Redshift 25

34 Tungsten Loading to Redshift Base Base Tables RedShift Tables Base Tables Transactions from Tungsten Master Tungsten Slave redshift Write data to CSV 3. Merge using SQL to base Staging Staging Tables Tables RedShift Staging Tables Remember this! Javascript load script e.g. hadoop.js CSV CSV Files CSV Files Files 1. Load to S3 using s3cmd S3 Storage 2. COPY to Redshift 25

35 Generate Schema Using ddlscan ddlscan Vertica Hadoop Redshift Tables Data types? Column lengths? Naming conventions? Staging tables? 26

36 Demo Setting up to MongoDB Setting up to Hadoop 27

37 Replicating from to generic applier 28

38 Use a generic applier Introducing the file applier a.k.a. the generic applier 29

39 Tungsten file applier Transactions from Tungsten Master Tungsten Slave GENERIC Write data to CSV Remember this? Javascript load script e.g. donothing.js CSV CSV Files CSV Files Files do something 30

40 Generic Replication in action # create table demo ( id int not null primary key, t varchar(30), ts timestamp); 31

41 Generic Replication in action # insert into demo values (1, 'hello world', null), (2, 'hello again', null), (3, 'bye', null); update demo set t = 'I am still here' where id = 2; 32

42 Generic Replication in action # select * from demo; id t ts hello world :59:20 2 I am still here :59:20 3 bye :59: rows in set (0.00 sec) 33

43 # CSV Generic Replication in action tungsten_opcode tungsten_seqno tungsten_row_id tungsten_commit_timestamp id t ts "I" "5" "2" " :54:46.000" "1" "hello world" " :54:46.000" "I" "6" "2" " :54:57.000" "2" "hello again" " :54:57.000" "I" "7" "2" " :55:05.000" "3" "bye" " :55:05.000" 34

44 # CSV Generic Replication in action tungsten_opcode tungsten_seqno tungsten_row_id tungsten_commit_timestamp id t ts "D" "8" "2" " :55:31.000" "2" \N \N "I" "8" "3" " :55:31.000" "2" "I am still here" " :55:31.000" 35

45 Setting up generic replication # MASTER --enable-batch-master=true \ --repl-svc-extractor-filters=schemachange 36

46 Setting up generic replication # SLAVE --batch-enabled=true \ --batch-load-language=js \ --enable-batch-slave=true \ --batch-load-template=donothing \ --repl-svc-applier-filters=monitorschemachange \ 37

47 Setting up generic replication # SLAVE - configure CSV format --property=... replicator.datasource.global.csv.fieldseparator= \ replicator.datasource.global.csv.useheaders=true \ replicator.datasource.global.csvtype=custom \ replicator.filter.monitorschemachange.notify=true \ 38

48 Setting up generic replication # All together tungsten-sandbox -m \ --topology=fileapplier \ --verbose 39

49 Demo Using a generic applier 40

50 Grow your own replicator Based on the file applier... and more! Replicate data to SQLite. 41

51 42 SQLite applier

52 Where to find the code tungsten-replicator.org for the basic replicator tungsten-sandbox: 43

53 Where Is Everything? Tungsten Replicator builds are available on code.google.com Replicator documentation is available on Continuent website Contact Continuent for support 44

54 In Conclusion Tungsten Replicator now supports realtime loading from to MOSTLY ANYTHING Continuent has a wealth of data loading features on tap so stay tuned! 45

55 Follow us on Tungsten Replicator: