Ibrahim Sallam Director of Development

Transcription

1 Ibrahim Sallam Director of Development

2 Graphs, what are they and why? Graph Data Management. Why do we need it? Problems in Distributed Graph How we solved the problems

3

4 Simple Graph Node <- > Node h"p://opte.org

5 [finding] Japanese restaurants in New York City liked by people from Japan a challenging Facebook query. "The value of any piece of information is only known when you can connect it with something else that arrives at a future point in time, CIA s CTO Ira "Gus" Hunt

6 Person? Building Work Live RR Visit Eat Shop

7 PLM (Product Lifecycle Mgmt) CRM, Sales & Marke.ng Network Mgmt, Telecom Intelligence (Government & Business) IG Finance Social Networks Research: Genomics Healthcare

8 Data structure for representing complicated relationships between entities. Wide applications in Bioinformatics. Social networks. Network management etc.

9

10 Everyone specializes Doctors, Lawyers, Bankers, Developers Why was data so normalized for so long! NoSQL is all about the data specialist Specializing in Distribution / deployment Physical data storage Logical data model Query mechanism

11 Navigate Ingest Update Massive graph data require efficient and intelligent tools to analyze and understand it.

12 Ingest Big/Fast data demands write performance Most NoSQL solutions allow you to scale writes by Partitioning the data Understanding your consistency requirements Allowing you to defer conflicts

13 Ingest App- 1 (E (Ingest 1 2 { V 1 V 21 }) ) App- 2 (E (Ingest 23 { V 2 V 32 }) ) App- 3 (Ingest V 3 ) InfiniteGraph ObjecSvity/DB Persistence Layer V 1 E 12 V 2 E 23 V 3

14 Ingest IG Core/API E 12 E 23 Target Containers E(1- >2) E(3- >1) E(2- >3) E(1- >2) E(3- >2) E(2- >3) E(2- >1) E(3- >1) E(2- >3) E(3- >1) E(1- >2) E(1- >2) E(2- >1) E(2- >3) E(3- >1) E(3- >2) Pipeline Containers Pipeline Agent C 1 C 2 C 3

15 Excellent for efficient use of page cache Able to maintain full database consistency Achieves highest ingest rate in distributed environments Almost always has highest perceived rate Trading Off : Eventual consistency in graph (connections) Updates are still atomic, isolated and durable but phased External agent performs graph building Ingest

16 Ingest Nodes and Edges per second clients 8 clients 1 client 2 clients 4 clients 8 clients client 2 clients # of Processes 4

17 Partitioning and Read Replicas easy right! ApplicaSon(s) Distributed API Processor Processor Processor Processor ParSSon 1 ParSSon 2 ParSSon 3 ParSSon...n

18 Navigate ApplicaSon(s) Distributed API Processor Processor Processor Processor ParSSon 1 ParSSon 2 ParSSon 3 ParSSon...n

19 Navigate Detect local hops and perform in memory traversal Send the partial path to the distributed processing to continue the navigation. Intelligently cache remote data when accessed frequently Route tasks to other hosts when it is optimal

20 Navigate ApplicaSon Distributed API Processor Processor P(A,B,C,D) ParSSon 1 ParSSon 2

21 Schema vs Schema-less Database religion InfiniteGraph supports schema, but does not restrict connection types between vertices We take advantage of the schema when pruning. Working on a hybrid support.

22 Navigate PaSent PrescripSon Drug Visit Physician Outcome Ingredient Complaint Allergy

23 Navigate GraphViews are extremely powerful Allow Big Data to appear small! Connection inference can lead to exponential gains in query performance Views are reusable between queries Built into the native kernel

24 Objectivity/DB is a proven foundation Building distributed databases since 1993 A complete database management system Concurrency, transactions, cache, schema, query, indexing It s a Graph Specialist! Simple but powerful API tailored for data navigation. Easy to configure distribution model

25 Physically co-locate closely related data Driven through a declarative placement model Dramatically speeds local reads Dr Blake Dr Quinn Mr CiSzen Primary Physician Dr Jones Dr Smith With Located Located Has Has With Located Located Sunny- vale At Visit Visit At San Jose Facility Facility Data Page(s) PaSent Data Page(s) Facility Data Page(s)

26 AddVertex() IG Core/API Customizable Placement Distributed Object and RelaSonship Persistence Layer HostA HostB HostC HostX Zone 1 Zone 2

27 Users ParSSoned Distributed DB (ohen Document / KV) External / Legacy Data Distributed Data Processing Plagorm TransformaSon \ MDM RDBMS Document Graph Database ApplicaSons Business

28 Distributed update. Update we are working on it.

29 Have we solved all the distributed graph problems? Not quite, but We Learned to Stop Worrying.

30