Building an Agile Big Data Infrastructure Have We Been Looking at Databases Wrong this Whole Time?

Transcription

1 Building an Agile Big Data Infrastructure Have We Been Looking at Databases Wrong this Whole Time? Presented by: Cory Isaacson, CEO CodeFutures Corporation Spring 2014

2 Introduction Who I am Cory Isaacson, CEO/CTO of CodeFutures Providers of scalable database technology Author of Software Pipelines Understanding Big Data (coming soon) Leaders in scalability, performance, high-availability and database solutions based on real-world experience with dozens of cloud-based and data center hosted applications social networking, gaming, data collection, mobile, analytics Objective is to provide useful experience you can apply to scaling (and managing) your database tier for Big Data especially for high volume applications

3 What I will cover Have we been looking at databases the wrong way? An Agile View of Big Data

4 Big Data what does it all mean?

5 Where is all this Big Data coming from?

6 All data is Relational Data has no meaning without relationships to other data a set of unmatched CUSTOMER_ORDER records with no CUSTOMER a Social Network that just shows random friends The world we live in is relational CUSTOMERs have CUSTOMER_ORDERs (1:m) DEVICEs produce DEVICE_MESSAGEs (1:m) MEETUPs have MEMBERs (1:m) PERSONs have NAMEs (1:1) Relationships translate directly to indexing, performance optimal performance is achieved when related data is stored in close proximity for fast retrieval

7 Data modeling 101 Data modeling is a vital skill and process for any software project Discover an Entity Discover a Relationship Discover Attributes New DBMS options make data modeling more critical then ever Flexible NoSQL data stores can store virtually any type of structure Lack of upfront data design will run any project into trouble (eventually) beware of the fast start leading to a train wreck

8 Who manages data relationships? An RDBMS helps still requires proper design, understanding of the relational model NoSQL/NewSQL stores its all up to you

9 Have we been looking at databases the wrong way?

10 Have we been looking at databases the wrong way?

11 Have we been looking at databases the wrong way?

12 A story in the life of a Game application

13 A story in the life of a Game application The game is growing fast Players want to see the Game instances they have played

14 A story in the life of a Game application

15 A story in the life of a Game application The game is big now Players want to see the other Players for a Game instance they have played

16 A story in the life of a Game application

17 A story in the life of a Game application The Game is really big now but it s not making $$$ Need to track all Player Actions in the game driving huge Big Data

18 A story in the life of a Game application Players want to see a Leader Board of the Top 100 Players

19 A story in the life of a Game application Players want to see a Leader Board for all Players Players want to be in Teams and have Leader Boards for that structure

20 Have we been looking at databases the wrong way?

21 An Agile View for Big Data

22 The Agile View of Big Data Create Distributed Views that are Eventually Consistent to manage a Polyglot DBMS environment Instead of looking at your database as a static repository Look at it as a series of Streams Views

23 The ideal View

24 10 Characteristics of an Agile Big Data Platform 1. Data is processed as a dynamic real-time stream. 2. Data input should be simple. 3. Dynamic views are the core construct. 4. View schemas exactly match application needs. 5. Persistence engines are pluggable, using the right tool for a given view, embracing a Polyglot approach. 6. View queries are fast and easy. 7. Views and streams must support transactions. 8. Views auto-scale according to optimal search criteria. 9. Views and streams must be fully reliable with hassle-free management. 10. An Agile Big Data Platform must seamlessly interoperate with existing database infrastructures.

25 MySQL to BigQuery Example

26 Questions/Answers Cory Isaacson CodeFutures Corporation