How graph databases started the multi-model revolution

Size: px

Start display at page:

Download "How graph databases started the multi-model revolution"

Silvester Morrison
8 years ago
Views:

1 How graph databases started the multi-model revolution Luca Garulli Author and QCon Sao Paulo - March 26, 2015

2 Welcome to Big Data 90% of the data in the world today has been created in the last two years alone. - IBM

3 Just Data Jill (Customer) Commodore Amiga 1200 (Product) Order #134 (Order) Luca (Provider) Bruno (Provider) Monitor 40 (Product) Mouse (Product)

4 Just Data Commodore Data Jill by itself has little Amiga 1200 (Customer) (Product) value, it s the relationship between data Order #134 that gives it Luca (Order) (Provider) Bruno (Provider) incredible value Monitor 40 (Product) Mouse (Product)

between data Order #134 that gives it Luca (Order)

5 Relationships give data meaning Jill (Customer) Commodore Amiga 1200 (Product) (Makes) (Has) Order #134 (Order) (Has) (Sells) Luca (Provider) Bruno (Provider) (Sells) (Has) Monitor 40 (Product) (Sells) Mouse (Product)

6 Top NoSQL categories Key/Value Databases Document Databases Column Databases Graph Databases

7 Top NoSQL categories Key/Value Databases Document Databases Column Databases Graph Databases

8 Why do most NoSQL products avoid managing relationships?

9 Joins is the Evil Customer CustomerAddress Address ID Name ID Address ID Location 10 John 11 John Milan 33 London 24 Mike Paris 28 Mike Is this familiar? 18 Madrid 44 Moscow

John 10 24 10 33 24 Milan 33 London 24 Mike 32 44

10 Why is the join so slow?

11 Index Lookup: how does it work? A- Z A- L M- Z Imagine an Address Book where we want to find Luca s phone number

12 Index Lookup: how does it work? A- Z A- L M- Z A- L M- Z A- D E- L M- R S- Z Index algorithms are all similar and based on balanced trees

13 Index Lookup: how does it work? A- Z A- L M- Z A- L M- Z A- D E- L M- R S- Z A- D E- L A- B C- D E- G H- L

14 Index Lookup: how does it work? A- Z A- L M- Z A- L M- Z A- D E- L M- R S- Z A- D E- L A- B C- D E- G H- L E- G H- L E- F G H- J K- L

15 Index Lookup: how does it work? A- Z A- B A- D C- D A- L A- D E- L E- G E- L H- L A- L M- Z Found! M- Z This lookup took 5 steps. S- Z With millions of indexed records, the tree depth could be 1000 s of levels! M- R E- G H- L E- F G H- J K- L Luca

16 Joins Kill Performance Customer CustomerAddress Address Joins are executed every time ID Name ID Address ID Location 10 Johnyou cross 10 relationships Milan 11 John London 24 Mike 28 Mike Querying million of records joining 3-4 tables could generate billions of combinations 18 Paris 18 Madrid 44 Moscow

Milan 11 John 10 33 33 London 24 Mike 28 Mike Querying million of records 32 44

17 This is why the database query performance suffers as the database increases in size O(Log N)

18 RDBMS performance on traversal DATABASE SIZE PERFORMANCE

19 In a world that s becoming more connected, we need a better way to store data and manage relationships Read: Data is important, but relationships are even more fundamental today

20 A graph database is any storage system that provides index-free adjacency - Marko Rodriguez (author of TinkerPop Blueprints)

21 Every developer knows the Relational Model, but who knows the Graph one?

22 Back to school: Graph Theory crash course

23 Basic Graph Luca Visited Sao Paulo

24 Property Graph Model* Vertices are directed Luca company: OrientTechnologies Visited on: 2015 Sao Paulo people: 12,000,000 Vertices and Edges can have properties * Graph- Model

25 1-N and N-M Relationships Visited on: 2015 Luca Worked Sao Paulo on: 2015 An Edge connects only 2 vertices Use multiple edges to represent 1- N and N- M relationships

26 Congrats! This is your diploma in «Graph Theory»

27 The Graph theory is so simple, yet so powerful

28 How does a true* Graph Database manage relationships? *a Graph layer on top of a DBMS doesn t qualify as a true GraphDB

29 Each element in the Graph has own immutable Record ID #22:11 Visited #13:55 Luca on: 2015 (Edge) #15:99 Sao Paulo (Vertex) (Vertex)

30 #13:55 Luca out = #22:11 out = #13:55 #22:11 Visited on: 2015 (Edge) in = #15:99 in = #22:11 #15:99 Sao Paulo (Vertex) (Vertex) Connections use persistent pointers

31 #13:55 Luca out = #22:11 out = #13:55 #22:11 Visited on: 2015 (Edge) in = #15:99 in = #22:11 #15:99 Sao Paulo (Vertex) (Vertex)

32 #13:55 Luca out = #22:11 out = #13:55 #22:11 Visited on: 2015 (Edge) in = #15:99 in = #22:11 #15:99 Sao Paulo (Vertex) (Vertex)

33 A Graph Database creates the relationship just once (when the edge is created) VS RDBMS computes the relationship every time you query a database

34 When you move from a RDBMS to a Graph Database you jump from a O(log N) speed to a near O(1) With a Graph Database, the traversing time is not affected by database size! This is huge in the BigData age

35 Graph Databases Easily Manage Complex Relationships Lives in John Thriller Pulp Fiction Theater B NYC Likes Comedy Mr Bean Theater A San Josè No costs to traverse relationships: Recommendation engines Social Applications Spatial Apps Master Data Management Information Clustering Theater C

36 GraphDB Database Quadrant Graph Relationships Complexity > Key Value Column Relational Document Data Complexity >

37 GraphDB Database Quadrant Graph These were 1st generation NoSQL Relationships Complexity > products, where each tool was Relational only good at a few use cases Document Column Key Value Data Complexity >

39 1st Generation NoSQL: Scenario Redis or Memcache (Key/Value) Application Neo4j (GraphDB) Primary DB Oracle (RDBMS) ETL MongoDB (DocDB)

40 1st Generation NoSQL: Fact In > 90% of use cases, NoSQL products are used as second DBMS

41 1st Generation NoSQL: Problems - No standard between NoSQL products - Multiple vendors = multiple skills - ETL + synchronization code is costly to write and maintain - Performance and Reliability is hard to predict Redis or Memcache (Key/Value) Application Neo4j (GraphDB) Oracle (RDBMS) ETL MongoDB (DocDB)

42 2nd Generation NoSQL is Multi-Model

43 What s Multi-Model DBMS? Key/Value Document Graph Object Multi Model represents the intersection of multiple models in just one product

44 What s Multi-Model DBMS? Multi Model represents the intersection of multiple models in just one product Key/Value - Just one product to learn and maintain - Just one vendor relationship to manage - No ETL, Document no synchronization required Graph - Performance and Reliability is easy to test from the beginning Object

45 Relationships give data meaning Jill (Customer) Commodore Amiga 1200 (Product) (Makes) (Has) Order #134 (Order) (Has) (Sells) Luca (Provider) Bruno (Provider) (Sells) (Has) Monitor 40 (Product) (Sells) 3 Wheel Mouse (Product)

46 Multi-Model domain schema Actor name: string surname: string Legenda: V Vertex Edge Inherits Makes Customer Provider Order number: int date: datetime Sells price: decimal Has price: decimal Product name: string qty: int

47 Vertices and Edges are Documents Jill { Customer", name : Jill, surname : Raggio, ` phone : , details : { city : London", tags : millennial } Makes Order General purpose solution: JSON Schema-less Schema-full Schema-hybrid Nested documents Rich indexing and querying Developer friendly

48 Polymorphic queries SELECT * FROM Customer Jill (Customer) SELECT * FROM Provider Bruno (Provider) Luca (Provider) SELECT * FROM Actor Bruno (Provider) Jill (Customer) Luca (Provider)

49 Multi-Model complex domains schema Legenda: V Vertex MusicTaste Likes Account Edge Inherits Band Genre Performs Plays Location

50 Multi-Model complex domains Jill (Account) (Likes) Indie (Genre) (Plays) (Likes) Snow Patrol (Band) (Likes) Luca (Account) (Likes) Rock (Genre) 123, 1st Street Austin, TX (Location) (Performs) April 7, pm-11.30pm

51 Multi-Model Database Quadrant Graph Multi-Model Relationships Complexity > Key Value Column Relational Document Data Complexity >

52 Multi-Model Solutions

53 There are a few DBMSs that claim to be Multi-Model, but they do not have a true Graph Engine. The Graph is only a layer on top of the engine. Under the hood they do JOINs, which means traversal time is affected by database size.

54 Meet OrientDB The First Ever Multi-Model Database Combining Flexibility of Documents with Connectedness of Graphs

55 With a true Graph, Document, Key/Value and Object Oriented engine

56 OrientDB features FEATURES ORIENTDB)) MONGODB NEO4J MYSQL) (RDBMS) Operational Database X X X Graph Database X X Document Database X X Object-Oriented Concepts X Schema-full, Schema-less, Schema mix X User and Role & Record Level Security X Record Level Locking X X X SQL X X ACID Transaction X X X Relationships (Linked Documents) X X X Custom Data Types X X X Embedded Documents X X Multi-Master Zero Configuration Replication X Sharding X X Server Side Functions X X X Native HTTP Rest/ JSON X X Embeddable with No Restrictions X

57 DEMO

58 API & Standards Support for TinkerPop standard for Graph DB: Gremlin language and Blueprints API SQL + extensions for graphs JDBC driver to connect any BI tool HTTP/JSON support Drivers in Java, Node.js, Python, PHP,.NET, Perl, C/C++ and more

59 Availability and Integrity C C C C C C C Master Node Multi-master Replication Master Node Atomic, Consistent, Isolated and Durable (ACID) multi-statement transactions

60 Scalability and Performance C C C C C C C Master Node Master Node Auto- Discovered Node Multi-Master Replication, Sharding and Auto- Discovery to Simplify Ops +200k Tps on Commodity Hardware

61 Some numbers 70+ Committers contributing to the product 1000s Users from SMBs to Fortune 10 Companies. 50,000 Downloads per Month from 200+ countries. 17+ Years of Research have been put in the product

62 A Bright Future Graph DBMS increased their popularity by 500% within the last 2 years Document DBMS are the 3 rd fastest growing category

63 Some of Our Customers

64 Get Started for Free OrientDB Community Edition is FREE for any purpose (Apache 2 license) OrientDB Enterprise is Free for Development Udemy Getting Started Training is and Free

65 Thank you. Ask your questions on Twitter for the Big Data Panel using #QCONBIGDATA Luca

Why NoSQL? Your database options in the new non- relational world. 2015 IBM Cloudant 1

Why NoSQL? Your database options in the new non- relational world 2015 IBM Cloudant 1 Table of Contents New types of apps are generating new types of data... 3 A brief history on NoSQL... 3 NoSQL s roots