How To Scale Big Data

Transcription

1 Real-time Big Data An Agile Approach Presented by: Cory Isaacson, CEO CodeFutures Corporation Fall 2014

2 Introduction Who I am Cory Isaacson, CEO/CTO of CodeFutures Providers of scalable database technology Author of Software Pipelines Understanding Big Data (coming soon) Leaders in scalability, performance, high-availability and database solutions based on real-world experience with dozens of cloud-based and data center hosted applications social networking, gaming, data collection, mobile, analytics Objective is to provide useful experience you can apply to scaling (and managing) your database tier for Big Data especially for high volume applications

3 What I will cover Scaling Big Data: The fundamentals Have we been looking at databases the wrong way? An Agile View of Big Data

4 What is Big Data? Big Data is when you have a problem with your database Your database is growing with inevitable performance degradation and management headaches Resulting in upset users business interruptions late nights and weekends at work When a single monolithic database won t cut it

5 What is Big Data?

6 Challenges of Big Data Databases grow and get slower Challenges apply regardless of hosting environment

7 Big Data what does it all mean?

8 Where is all this Big Data coming from?

9 BigDoor

10 All CPUs wait at the same speed The I/O Barrier

11 Why databases slow down

12 Enemies of database performance Enemy #1: Table scans Lack of indexes causes: table scans Non-tested access queries causes: table scans

13 Enemy #1: Table Scan

14 Enemy #1: Table Scan

15 Enemies of database performance Enemy #2: Concurrency contention Poor data model design causes: excessive locking Transactions that are too long causes: excessive locking Single-threaded engine

16 Enemy #2: Concurrency Contention

17 Enemy #2: Concurrency Contention

18 Enemy #3: Slow writes Enemy #3: Slow writes Too many indexes in a single table causes: slow writes Table is too large causes: slow writes

19 Enemy #3: Slow Writes

20 Challenges apply to all types of databases Traditional RDBMS (MySQL, Postgres, Oracle ) I/O bound Multi-user, lock contention High-availability NoSQL, NewSQL, Column Databases Reliability is decent if replication enabled n usually at the cost of performance Limits of a single server n and sometimes a single thread Data dumps to disk High-availability

21 The Laws of Databases Law #1: Small Databases are fast Law #2: Big Databases are slow Law #3: Keep databases small

22 What is the answer? Database sharding is the only effective method for achieving scale, elasticity, reliability and easy management regardless of your database technology

23 What is Database Sharding? Horizontal partitioning is a database design principle whereby rows of a database table are held separately... Each partition forms part of a shard, which may in turn be located on a separate database server or physical location. Wikipedia

24 What is Database Sharding? Start with a big monolithic database break it into smaller databases across many servers using a Shard Key

25 The key to Database Sharding

26 Database Sharding Architecture

27 Database Sharding the results

28 Breaking the Database Performance Barrier

29 All data is Relational Data has no meaning without relationships to other data a set of unmatched CUSTOMER_ORDER records with no CUSTOMER a Social Network that just shows random friends The world we live in is relational CUSTOMERs have CUSTOMER_ORDERs (1:m) DEVICEs produce DEVICE_MESSAGEs (1:m) MEETUPs have MEMBERs (1:m) PERSONs have NAMEs (1:1) Relationships translate directly to indexing, performance optimal performance is achieved when related data is stored in close proximity for fast retrieval

30 Data modeling 101 Data modeling is a vital skill and process for any software project Discover an Entity Discover a Relationship Discover Attributes New DBMS options make data modeling more critical then ever Flexible NoSQL data stores can store virtually any type of structure Lack of upfront data design will run any project into trouble (eventually) beware of the fast start leading to a train wreck

31 Who manages data relationships? An RDBMS helps still requires proper design, understanding of the relational model NoSQL/NewSQL stores its all up to you

32 The 2 Types of Database Sharding Blackbox Sharding attempts to evenly shard across all available servers no developer visibility or control can work acceptably for simple NoSQL data stores easily supports single-row/object results Relational Sharding selective sharding of large data explicit developer control and visibility more efficient for result sets and searchable queries Both utilize sharding on a data key typically modulus, range or consistent hash

33 How Database Sharding works for NoSQL

34 How Relational Sharding works Primary Shard Table Shard Child Tables Global Tables

35 How Relational Sharding works

36 How Relational Sharding works Shard key recognition in SQL SELECT * FROM customer WHERE customer_id = 1234 INSERT INTO customer (customer_id, first_name, last_name, addr_line1, ) VALUES (2345, John, Jones, 123 B Street, ) UPDATE customer SET addr_line1 = 456 C Avenue WHERE customer_id = 4567

37 The good, the bad and the ugly about a Distributed Big Data architecture The good distributed shard-nothing single-server low-latency operations are fast The bad distributed distributed operations are inherently slow n distributed Transactions n Shuffle Joins, scatter-gather n the network becomes the bottleneck The ugly data rebalancing intensive process, lot of overhead

38 What about Multi-Shard result sets?

39 Database Sharding The End Game

40 Shard Actions Shard Actions in order of performance Shard Read n read from a single shard Shard Write n write to a single shard Multi-Shard Read (Go Fish) n read/aggregate/sort from multiple shards Multi-Shard Write n distributed writes/transactions

41 Database Sharding the End Game The objective for any sharded environment The highest possible percentage of low-latency single server operations n Shard Read, Shard Write The fewest possible occurrences of Multi-Shard Read Avoid Multi-Shard Write As many database operations against a single shard server as possible Relational Sharding can be ideal for many applications natural co-location of related data that is meaningful to the application

42 Have we been looking at databases the wrong way?

43 Have we been looking at databases the wrong way?

44 Have we been looking at databases the wrong way?

45 A story in the life of a Game application

46 A story in the life of a Game application The game is growing fast Players want to see the Game instances they have played

47 A story in the life of a Game application

48 A story in the life of a Game application The game is big now Players want to see the other Players for a Game instance they have played

49 A story in the life of a Game application

50 A story in the life of a Game application The Game is really big now but it s not making $$$ Need to track all Player Actions in the game driving huge Big Data

51 A story in the life of a Game application Players want to see a Leader Board of the Top 100 Players

52 A story in the life of a Game application Players want to see a Leader Board for all Players Players want to be in Teams and have Leader Boards for that structure

53 Have we been looking at databases the wrong way?

54 The Agile View of Big Data

55 An Agile View for Big Data Wouldn t it be nice if we could work with data in a continuous agile Stream support agile Views of the data as needed for application requirements maintain data integrity across Polygot DBMS engines if required isolate complex data structures from the application logic get it back to simple reads and writes

56 An Agile View for Big Data

57 The Agile View of Big Data Create Distributed Views that are Eventually Consistent to manage a Polyglot DBMS environment Instead of looking at your database as a static repository Look at it as a series of Streams Views

58 The ideal View

59 10 Characteristics of an Agile Big Data Platform 1. Data is processed as a dynamic real-time stream. 2. Data input should be simple. 3. Dynamic views are the core construct. 4. View schemas exactly match application needs. 5. Persistence engines are pluggable, using the right tool for a given view, embracing a Polyglot approach. 6. View queries are fast and easy. 7. Views and streams must support transactions. 8. Views auto-scale according to optimal search criteria. 9. Views and streams must be fully reliable with hassle-free management. 10. An Agile Big Data Platform must seamlessly interoperate with existing database infrastructures.

60 MySQL to BigQuery Example

61 Questions/Answers Cory Isaacson CodeFutures Corporation