Introduction to Polyglot Persistence. Antonios Giannopoulos Database Administrator at ObjectRocket by Rackspace

Size: px

Start display at page:

Download "Introduction to Polyglot Persistence. Antonios Giannopoulos Database Administrator at ObjectRocket by Rackspace"

Jordan Rodgers
7 years ago
Views:

1 Introduction to Polyglot Persistence Antonios Giannopoulos Database Administrator at ObjectRocket by Rackspace FOSSCOMM 2016

2 Background - 14 years in databases and system engineering - NoSQL ObjectRocket by Rackspace - Passionate about MongoDB & Cassandra

3 What is Polyglot Persistence? A set of applications that use several core database technologies Application Layer Relational Database Key/Value Store Column-family Database

4 What is Polyglot Persistence? Using the right tool for the right use case

5 Let there be RDBMS Monoglot was (and still is) fine for simple application (one type of workload) But applications become complex A simple E-commerce platform must have: - Session data (Add to Basket) - Search Engine (Search for products) - Recommendation engine (Customers Who Bought This Item Also Bought) - Payment platform - Geo Location service

6 Applications growing rapidly

7 A RDBMS tale In the good old days monoglot = RDBMS Once upon a time there was an RDBMS with performance issues - Vertical scaling - Secondary indexes - Partitioning - Denormalize - Read-Only Slaves - Shading - Start separating workloads But The more we scale the more features we miss

8 The 3V Era Volume: Amount of data Velocity: Speed of data processing Variety: Number of types of data New databases introduced for Big Data General-purpose DB is no longer on-trend Per use-case Datastores becoming more popular The raise of Polyglot Persistence

9 CAP theorem

10 Picking the right tools Data Structure - Relational databases (Oracle, MySQL) - Key-value stores (Redis, Riak) - Column Family stores (Cassandra, Hbase) - Document databases (MongoDB, CouchDB) - Graph databases (Neo4J)

11 Relational Databases Based on the relational model (Codd) Data are organized in tables (Row and columns) Using SQL (Structured Query Language)

12 Relational Databases Use when: - Your dataset is relational - Strong consistency needed - Access patterns are unknown But Doesn t scale well horizontally Use Cases: - Due to early adoption are everywhere - Payment systems

13 Key/Value stores A big hash map associative array - Very simple, One key <-> One value - Very fast read/write - No secondary indexes { Key : (VRN)} => {value (car facts)} [make#ford {YYY0000} => model#fiesta year#2010]

14 Key/Value stores Use when: - Operations are based on the key - Data is not highly related - Basic CRUD needed But Complex queries are painful Use cases: - Session Data - User Profile/Preferences - Shopping cards

15 Document Databases Nested structures of keys and their values - Very flexible schema (JSON, XML) - One key One value but value is visible to queries - Supports hierarchical data - Supports secondary indexes { id (VRN)} => { document (car facts)} { make : Ford, {YYY0000} => model : Fiesta, year :2010 }

16 Document Databases Use when: - You don t know much for the schema - Unstructured and Heterogeneous data But Joins and references are tricky De-normalization requires more space Use cases: - Product Catalog - CMS - Event logging from different sources

17 Column family In a table, data of the same column is stored together (K-V that V is K-V) - Data organized as columns - Great for sparse tables - Very fast column operation including aggregation { id (VRN)} => { column families (car facts)} { car :{ make : Ford, model : Fiesta }, {YYY0000} => parts :{ }, service : { } }

18 Use when: - Big Data (Huge write volumes) - Versioning (Time-series data) But Know your statements in advance Schema design is not trivial Use cases: - Time series data - Bidding platforms - Playlists Column family

19 Graph Databases Inspired by the graph theory Nodes hold Data/Entities Each node connect to others using attribute(s)

20 Graph Databases Use when: - Highly interconnected data - Define explicit relationships and need traversal queries But Doesn t scale well horizontally Use Cases: - Where 3 rd degree (or higher) relationship needed - Social Media - Queries like friend of a friend

21 What about Hadoop? Framework for distributed storing and processing large sets of data Use for: - ETL - Read raw data, apply filters create structured summary - Exploration engine / Data discovery - Data archive / Massive Storage But Do not use Hadoop as a database replacement

22 Picking the right tools - Questions Data structure: - Does it have a natural structure? Is unstructured? - How it is connected to each other? - How is it distributed? - How much data? Access Patterns: - Read/Write ratio? - Uniform or random? - What is more important?

23 Picking the right tools - Questions Organization needs: - Do I need Authentication? What type? - Do I need Encryption? - Do I need Backups? - Do I need a Disaster Site? - What level of Monitoring? - Drivers, Languages? Tools - Third Party tools - Add-ons/Plugins

24 Picking the right tools - Questions Maturity - How long is in the market? Documentation - Books, tutorials - Training Type of Support - Community - Commercial support

25 Challenges Define the architecture - Decide which datastore will use to store certain data - Wrong decision can lead to painful migration(s) Deployment complexity - Provision different type of HW, OS, patches - Backup/Restore - Control configuration changes - Monitor all different components

26 Challenges Application complexity - Different connection per datastore - Handle different type of errors - Map different results on the application layer - Keep Datastores in-sync (cross-database consistency) - Active/Passive topology - Active/Active topology Training for Devs and Ops - Develop new skills for your teams - Support for a period of time

27 Don ts and anti-patterns Over engineering - Keep it simple - Remove pieces that don t add value Conform to stereotypes - Use cases are general guidelines - Benchmarks are indicators Stay static - Try new technologies - Being on-trend but cautious

28 Use Case: Mongo ElasticSearch MongoDB is using B-Tree indexes for everything B-Tree is great but not excel for all use-cases Start getting hard limits for certain use cases like FullText search Lucene engine is better option for Full text search (using inverted indexes) We already had the ElasticSearch to our portfolio so we just connected the two products

29 Use Case: Mongo Elastic Search - Connector is using the Active/Passive model - Writes must go through MongoDB and then propagated to ElasticSearch - Flexible, user can pick what to propagate: - Database(s) - Collection(s) - Field mapping - Indexing/analyzing

30 Use Case: Mongo Elastic Search - An initial sync is needed (bulk API) - Connector is using a tailable cursor that reads MongoDB oplog and propagate the changes - Similar to Extract, Transform, Load (ETL) - Application is responsible to properly direct the read requests to the most suitable datastore

31 Call me Polyglot, Call me Multi-Model MongoDB (version 3 or higher) - MMAPv1 - WiredTiger - In-Memory Percona Server for MongoDB - MMAPv1 - WiredTiger - RocksDB - PerconaFT

32 The future(?) Multi Model databases Support different models within the engine -OR- Offer different layers on top of the engine OrientDB supporting graph, document and key/value models Relationships are managed as in graph databases with direct connections between records FoundationDB feature layers on top of a key-value store

33 Keep in touch : I iamantonios antonios.giannopoulos@rackspace.co.uk We are hiring!!! Data Engineers, DevOps, DBAs and more

34 Questions? Thank you!!!

Data Services Advisory

Data Services Advisory Modern Datastores An Introduction Created by: Strategy and Transformation Services Modified Date: 8/27/2014 Classification: DRAFT SAFE HARBOR STATEMENT This presentation contains