Big Data with MySQL. Percona Live Santa Clara Ivan Zoratti V Friday, 3 May 13

Transcription

1 Big Data with MySQL Percona Live Santa Clara 2013 Ivan Zoratti V

2 Who is Ivan?

3 SkySQL Leading provider of open source databases, services and solutions Home for the founders and the original developers of the core of MySQL The creators of MariaDB, the drop-off, innovative replacement of MySQL

4 What is Big Data?

5 Big Data! Big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. 5

6 Big Data By Structure Unstructured Store everything you have/you find In any format and shape You do not know how to use it, but it may come handy Storing unstructured data is usually cheaper than storing it in a more structured datastore Does not fit well in a relational database Examples: Text: Plain text, documents, web content, messages Bitmap: Image, audio, video Typical approach: Mining, pattern recognition, tagging Usually batch analysis Structured Store only what you need In a good format, ready to be used You should already know how to use it, or at least what it means Storing structured data is quite expensive Raw data, indexing, denormalisation, aggregation A relational database is still the best choice Examples: Machine-Generated Data (MGD) Tags, counters, sales Typical approach: BI tools, reporting Real time analysis change data capture 6

7 Big Data By Structure Unstructured Store everything you have/you find In any format and shape You do not know how to use it, but it may come handy Storing unstructured data is usually cheaper than storing it in a more structured datastore Does not fit well in a relational database Examples: Text: Plain text, documents, web content, messages Bitmap: Image, audio, video Typical approach: Mining, pattern recognition, tagging Usually batch analysis Structured Store only what you need In a good format, ready to be used You should already know how to use it, or at least what it means Storing structured data is quite expensive Raw data, indexing, denormalisation, aggregation A relational database is still the best choice Examples: Machine-Generated Data (MGD) Tags, counters, sales Typical approach: BI tools, reporting Real time analysis change data capture 7

8 How Big is Big Data? Data Factors Size Speed to collect/ generate Variety Resources Administrators Developers Growth Collection Processing Availability To whom? For how long? In which format? Aggregated Detailed Infrastructure 8

9 How to manage Big Data Collection - Storage - Archive Load - Transform - Analyze Access - Explore - Utilize 9

10 Big Data with MySQL

11 Technologies to Use / Consider / Watch MyISAM and MyISAM compression Cassandra Storage Engine InnoDB compression Connect Storage Engine MySQL 5.6 Partitioning MariaDB Optimizer Columnar Databases InfiniDB Infobright MariaDB Virtual & Dynamic Columns TokuDB Storage Engine 11

12 Columnar Databases Automatic compression Automatic column storage Data distribution Map/Reduce approach MPP / Parallel loading No indexes On public clouds, HW or SW appliances 12

13 TokuDB Increased Performance Increased Compression Online administration No Index rebuild 13

14 MyISAM Static, dynamic and compressed format Multiple key cache, CACHE INDEX and LOAD INDEX Compressed tables Horizontal partitioning (manual) External locking 14

15 InnoDB/XtraDB Data Load Pre-order data Split data into chunks unique_checks = 0; foreign_key_checks = 0; sql_log_bin = 0; innodb_autoinc_lock_mode = 2; Compression and block size SET GLOBAL innodb_file_per_table = 1; SET GLOBAL innodb_file_format = Barracuda; CREATE TABLE t1 ( c1 INT PRIMARY KEY, c2 VARCHAR(255) ) ROW_FORMAT = COMPRESSED KEY_BLOCK_SIZE = 8; LOAD DATA LOCAL INFILE '/usr2/t1_01_simple' INTO TABLE t1; Query OK, rows affected (1 hour 34 min 7.49 sec) Records: Deleted: 0 Skipped: 0 Warnings: 0 LOAD DATA LOCAL INFILE '/usr2/t1_01_simple' INTO TABLE t2; Query OK, rows affected (25 min sec) Records: Deleted: 0 Skipped: 0 Warnings: 0 Persistent optimizer stats innodb_stats_persistent innodb_stats_auto_recalc 15

16 Partitioning (MySQL 5.6) Partitioning Types RANGE, LIST, RANGE COLUMN, HASH, LINEAR HASH, KEY LINEAR KEY, sub-partitions Partition and lock pruning Use of INDEX and DATA DIRECTORY PARTITION ADD, DROP, REORGANIZE, COALESCE, TRUNCATE, EXCHANGE, REBUILD, OPTIMIZE, CHECK, ANALYZE, REPAIR CREATE TABLE t1 ( c1 INT, c2 DATE ) PARTITION BY RANGE( YEAR( c2 ) ) SUBPARTITION BY HASH ( TO_DAYS( c2 ) ) ( PARTITION p0 VALUES LESS THAN (1990) ( SUBPARTITION s0 DATA DIRECTORY = '/disk0/data' INDEX DIRECTORY = '/disk0/idx', SUBPARTITION s1 DATA DIRECTORY = '/disk1/data' INDEX DIRECTORY = '/disk1/idx' ),... ALTER TABLE t1 EXCHANGE PARTITION p3 WITH TABLE t2; -- Range and List partitions ALTER TABLE t1 REORGANIZE PARTITION p0,p1,p2,p3 INTO ( PARTITION m0 VALUES LESS THAN (1980), PARTITION m1 VALUES LESS THAN (2000)); -- Hash and Key partitions ALTER TABLE t1 COALESCE PARTITION 10; ALTER TABLE t1 ADD PARTITION PARTITIONS 5; 16

17 MariaDB Optimizer Multi-Range Read (MRR) * ORDER BY... LIMIT <limit> * MariaDB 10 Index Merge / Sort intersection Batch Key Access * Block hash join Subqueries Semi-join * Materialization * subquery cache Cost-based choice of range vs. index_merge LIMIT... ROWS EXAMINED <limit> (*) - Available in MySQL

18 Virtual & Dynamic Columns VIRTUAL COLUMNS For InnoDB, MyISAM and Aria PERSISTENT (stored) or VIRTUAL (generated) CREATE TABLE t1 ( c1 INT NOT NULL, c2 VARCHAR(32), c3 INT AS ( c1 MOD 10 ) VIRTUAL, c4 VARCHAR(5) AS ( LEFT(B,5) ) PERSISTENT); DYNAMIC COLUMNS Implement a schemaless, document store COLUMN_ CREATE, ADD, GET, LIST, JSON, EXISTS, CHECK, DELETE Nested colums are allowed Main datatypes are allowed Max 1GB documents CREATE TABLE assets ( item_name VARCHAR(32) PRIMARY KEY, dynamic_cols BLOB ); INSERT INTO assets VALUES ( 'MariaDB T-shirt', COLUMN_CREATE( 'color', 'blue', 'size', 'XL' ) ); INSERT INTO assets VALUES ( 'Thinkpad Laptop', COLUMN_CREATE( 'color', 'black', 'price', 500 ) ); 18

19 Cassandra Storage Engine Column Family == Table Rowkey, static and dynamic columns allowed Batch key access support SET cassandra_default_thrift_host = ' ' CREATE TABLE cassandra_tbl ( rowkey INT PRIMARY KEY, col1 VARCHAR(25), col2 BIGINT, dyn_cols BLOB DYNAMIC_COLUMN_STORAGE = yes ) ENGINE = cassandra KEYSPACE = 'cassandra_key_space' COLUMN_FAMILY = 'column_family_name'; 19

20 Connect Storage Engine Any file format as MySQL TABLE: ODBC Text, XML, *ML Excel, Access etc. MariaDB CREATE TABLE options Multi-file table Table Autocreation CREATE TABLE handout ENGINE = CONNECT TABLE_TYPE = XML FILE_NAME = 'handout.htm' HEADER = yes OPTION_LIST = 'name = TABLE, coltype = HTML, attribute = (border=1;cellpadding=5)'; Condition push down Read/Write and Multi Storage Engine Join CREATE INDEX 20

21 Starting Your Big Data Project

22 Why would you use MySQL? Time Knowledge Infrastructure Costs Simplified Integration Not so big data 22

23 Apache Hadoop & Friends Mahout PIG HIVE Ambari, Ganglia, Nagios HCatalog HBASE ZooKeeper Sqoop Cascading Oozie Flume MapReduce Protobuf, Avro, Thrift Fuse-DFS Chukwa HDFS Cassandra 23

24 MySQL & Friends Scripts Stored Procedures DML DB Schema / DDL MySQL/MariaDB SkySQL DS Mahout SDS, Ganglia, Nagios mysqlimport Cascading Talend, Pentaho Connect SQL Optimizer MySQL/MariaDB/Storage Engines 24

25 Join us at the Solutions Day Cassandra and Connect Storage Engine Map/Reduce approach - Proxy optimisation Multiple protocols and more 25

26 Thank You! izoratti.blogspot.com