Graph Databases What makes them Different?

Transcription

1 Graph Databases What makes them Different? Darren Wood Chief Architect, InfiniteGraph

2 NoSQL Data Specialists Everyone specializes Doctors, Lawyers, Bankers, Developers Why was data so normalized for so long! NoSQL is all about the data specialist Specializing in Distribution / deployment Physical data storage Logical data model Query mechanism

3 Polyglot NoSQL Architectures Users ParAAoned Distributed DB (ocen Document / KV) External / Legacy Data Distributed Data Processing Pla;orm TransformaAon \ MDM RDBMS Document Graph Database ApplicaAons Business

4 The Physical Data Model Becoming a relationship specialist Rows/Columns/Tables Relationship/Graph Optimized MeeAngs P1 P2 Place Time Alice Bob Denver Alice Met Charlie Calls From To Time DuraAon Bob Carlos 13:20 25 Bob Charlie 17:10 15 Bob Called 13:20 Called 17:10 Payments From To Date Amount Carlos Charlie Carlos Paid

5 Navigational Query Performance

6 Scaling Writes Big/Fast data demands write performance Most NoSQL soluaons allow you to scale writes by ParAAoning the data Understanding your consistency requirements Allowing you to defer conflicts

7 Scaling Graph Writes ACID Transactions App- 1 (E (Ingest 1 2 { V 1 V 21 }) ) App- 2 (E (Ingest 23 { V 2 V 32 }) ) App- 3 App- 3 (Ingest V 3 ) InfiniteGraph ObjecAvity/DB Persistence Layer V 1 E 12 V 2 E 23 V 3

8 High Performance Edge Ingest E23 E(1- >2) E(2- >3) E(3- >1) E(2- >1) E(2- >3) E(1- >2) E(1- >2) E(2- >3) E(3- >1) E(1- >2) E(3- >2) E(2- >1) E(2- >3) E(3- >1) E(3- >1) E(3- >2) Pipeline C1 Pipeline Containers E12 Target Containers IG Core/API C2 Agent C3

9 Trade offs Excellent for efficient use of page cache Able to maintain full base data consistency Achieves highest ingest rate in distributed environments Almost always has highest perceived rate Trading Off : Eventual consistency in graph Updates are still atomic, isolated and durable but phased External agent performs graph building

10 Result Nodes and Edges per second clients Hosts 4 4 clients Hosts 1 client 2 clients 4 clients 8 clients clients Hosts 1 Single client Host 4

11 Scaling Reads and Query Partitioning and Read Replicas easy right! ApplicaAon(s) Distributed API Processor Processor Processor Processor ParAAon 1 ParAAon 2 ParAAon 3 ParAAon...n

12 Why are Graphs Different? ApplicaAon(s) Distributed API Processor Processor Processor Processor ParAAon 1 ParAAon 2 ParAAon 3 ParAAon...n

13 Optimizing Distributed Navigation Pregel Clones Message passing for each hop, messages processed at the data host for the target vertex High concurrency, no data leaves its physical host Message marshalling and transport is expensive Distributed Caching Models Essentially trying to cache graph in memory over multiple hosts Requires too much memory for large graphs Issues with cache consistency

14 Optimizing Distributed Navigation InfiniteGraph both Detect local hops and perform in memory traversal Intelligently cache remote data when accessed frequently Route tasks to other hosts when it is optimal ApplicaAon Distributed API Processor Processor P(A,B,C,D) ParAAon 1 ParAAon 2

15 Schema It s not your enemy! (at least not all the Ame...) Schema vs Schema-less Database religion No time for a full debate here InfiniteGraph supports schema, but does not restrict connection types between vertices Planning to also support Document Style Nodes

16 GraphViews Leveraging Schema in the Graph PaAent PrescripAon Drug Visit Physician Outcome Ingredient Complaint Allergy

17 Schema Enables Views GraphViews are extremely powerful Allow Big Data to appear small! Connection inference can lead to exponential gains in query performance Views are reusable between queries Built into the native kernel

18 Advanced Configured Placement Physically co-locate closely related data Driven through a declarative placement model Dramatically speeds local reads Dr Quinn Dr Blake Primary Physician Mr CiAzen Dr Smith Dr Jones With Located Has Located Sunny- vale Facility Data Page(s) At Visit Has Visit PaAent Data Page(s) With Located At Located San Jose Facility Facility Data Page(s)

19 Why InfiniteGraph? Objectivity/DB is a proven foundation Building distributed databases since 1993 A complete database management system Concurrency, transactions, cache, schema, query, indexing It s a Graph Specialist! Simple but powerful API tailored for navigation of data Easy to configure distribution model

20 Fully Distributed Data Model AddVertex() IG Core/API Customizable Placement Distributed Object and RelaAonship Persistence Layer HostA HostB HostC HostX Zone 1 Zone 2

21 InfiniteGraph is a Complete Database InfiniteGraph helps manage the things you don t want to do, but want to have done: Concurrency TransacAons (commit/rollback) Controlled mula- user reading during updates Schema Control Build complex data structures, make changes easily and migrate exisang data Distribu9on Sharing large amounts of distributed data between distributed processes Indexes Choose built- in key- value, b- tree or other indexes Cache Keep large secaons of the graphs in configurable memory caches

22 Super Simple API Person alice = new Person( Alice ); hellographdb.addvertex( alice ); Person bob = new Person( Bob ); hellographdb.addvertex( bob ); Person carlos = new Person( Carlos ); hellographdb.addvertex( carlos ); Person charlie = new Person( Charlie ); hellographdb.addvertex( charlie );

23 Adding Edges MyEdgeType edge = new MyEdgeType(); vertexa.addedge ( edge, vertexb, EdgeKind.??? ); Meeting denvermeeting = new Meeting("Denver", " "); alice.addedge(denvermeeting, bob, EdgeKind.BIDIRECTIONAL); Call bobcalltocarlos = new Call(getRandomJulyTime()); bob.addedge(bobcalltocarlos, carlos, EdgeKind.BIDIRECTIONAL); Payment payment = new Payment( ); carlos.addedge(payment, charlie, EdgeKind.BIDIRECTIONAL); Call bobcalltocharlie = new Call(getRandomJulyTime()); bob.addedge(bobcalltocharlie, charlie, EdgeKind.BIDIRECTIONAL);

24 The Result

25 Graph Traversal (Navigation) Queries Use an instance of the Navigator class to perform a navigation query. A navigation instance is highly customizable, but is comprised of the following basic parts: Origin : The vertex from which to begin Guide strategy Guide.Strategy.SIMPLE_BREADTH_FIRST Guide.Strategy.SIMPLE_DEPTH_FIRST Graph Views : Powerful filtering and Qualifiers Qualifying valid intermediate paths and results Handlers A result handler

26 Tools To Suit the Solution

27 Practical Applications

28 Pathfinding Intelligence, Police, Counter Terrorism Financial Transactions and Fraud Discover Paths Between Entities One to One, Many to Many Constrain by Edge and Vertex Type Edge and Vertex Attributes (including temporal) Required Sequence of Events Length (Hops), Total Edge Weight

29 Graph Analysis (Algorithms) Bob Degree Centrality Sam Closeness and Betweeness Centrality Fred

30 Graph Analysis (Algorithms) Social Networks Most connected participants Influencers Important Syndicates or Sub-networks Central figures in crime organisations Business Intelligence Discovering Knowledge Assets Complex Big Data Analytics

31 Graph Analysis (Patterns) Crime (again) Recognize common patterns of activity Complex chains of interaction Security Recognize attack/threat patterns Auditing / log analytics Targeting Advertising To specific browsing patterns

32 Many Many More. Spatial data Financial Services Defense / Situational Awareness Sciences Health Care Genealogy Logistics Tracking PLM