An Introduction To Presented by Leon Guzenda, Founder, Objectivity

Transcription

1 An Introduction To Graph Databases Presented by Leon Guzenda, Founder, Objectivity Mark Maagdenberg, Sr. Sales Engineer, Objectivity Paul DeWolf, Dir. Field Engineering, Objectivity August 21, 2012

2 Overview Introductions Graph Theory Commonly Used Graph Algorithms Graph Databases Current Implementations Use Cases Hands-On Tutorial

3 We Are From Objectivity Inc. Company Objectivity, Inc. is headquartered in Sunnyvale, CA. Established in 1988 to tackle database problems that network/hierarchical/relational and file-based technologies struggle with. Objectivity has over two decades of Big Data and NoSQL experience Products Develops NoSQL platforms for managing and discovering relationships and patterns in complex data: Objectivity/DB - an object database that manages localized, centralized or distributed databases InfiniteGraph - a massively scalable graph database built on Objectivity/DB that enables organizations to find, store and exploit the relationships in their data Markets The Big Data market is projected to be around $12B in 2012, with a CAGR of 28% over the next five years. 40% per year data growth, cloud adoption, mobile usage and improved real-time analytics underpin Objectivity s growth opportunities as a Big Data analytics enabler. Customers Embedded in hundreds of enterprises, government organizations and products - millions of deployments. Financials Consistently generates increased revenues. Pi Privately held ldby the employees and a few venture capital companies. Copyright Objectivity, Inc. 2012

4 GRAPH THEORY

5 The History of Graph Theory 1736: Leonard Euler writes a paper on the Seven Bridges of Konisberg 1845: Gustav Kirchoff publishes his electrical circuit laws 1852: Francis Guthrie poses the Four Color Problem 1878: Sylvester publishes an article in Nature magazine that describes graphs 1936: Dénes Kőnig publishes a textbook on Graph Theory 1941: Ramsey and Turán define Extremal Graph Theory 1959: De Bruijn publishes a paper summarizing Enumerative Graph Theory 1959: Erdos, Renyi and Gilbert define Random Graph Theory 1969: Heinrich Heesch solves the Four Color problem 2003: Commercial Graph Database products start appearing on the market

6 Graph Theory Terminology... VERTEX: A single node in a graph data structure EDGE: A connection between a pair of VERTICES PROPERTIES: Data items that belong to a particular Vertex WEIGHT: A quantity associated with a particular Edge GRAPH: A collection of linked Vertex and Edge objects Vertex 1 Vertex 2 Edge 1 City: San Francisco Pop: 812,826 Road: I-101 Miles: 47.8 City: San Jose Pop: 967,487

7 ...Graph Theory Terminology... SIMPLE/UNDIRECTED GRAPH: A Graph where each VERTEX may be linked to one or more Vertex objects via Edge objects and each Edge object is connected to exactly two Vertex objects. Furthermore, neither Vertex connected to an Edge is more significant than the other. DIRECTED GRAPH: A Simple/Undirected Graph where one Vertex in a Vertex + Edge + Vertex group (an Arc or Path ) can be considered d the Head of the Path and the other can be considered the Tail. MIXED GRAPH: A Graph in which some paths are Undirected and others are MIXED GRAPH: A Graph in which some paths are Undirected and others are Directed.

8 ...Graph Theory Terminology LOOP: An Edge that is doubly-linked to the same Vertex MULTIGRAPH: A Graph that allows multiple Edges and Loops QUIVER: A Graph where Vertices are allowed to be connected by multiple Arcs. A Quiver may include Loops. WEIGHTED GRAPH: A Graph where a quantity is assigned to an Edge, e.g. a Length assigned to an Edge representing a road between two Vertices representing cities. HALF EDGE: An Edge that is only connected to a single Vertex LOOSE EDGE: An Edge that isn't connected to any Vertices. CONNECTIVITY: Two Vertices are Connected if it is possible to find a path between them.

9 COMMONLY USED GRAPH ALGORITHMS Mac Evans

10 Commonly Used Graph Algorithms... CONNECTEDNESS: Check whether or not a set of nodes in a Graph are connected. All of the nodes in the graph below are connected, e.g. A to B, A to C via B etc. SHORTEST PATH: The path between two nodes that visits the fewest intermediate nodes. In the graph above, A->B->C->D is shorter than A->B->C->B->D (disallowing loops) NODE DEGREE: The degree of a node in a network is a count of the number of connections it has to other nodes. The degree distribution is the probability distribution of these degrees in the whole network. In the graph below, A and D have a node degree of 1. B and C have a node degree of 3.

11 ...Commonly Used Graph Algorithms... CENTRALITY: An assessment of the importance of a node within a network. Degree Centrality is the simplest, being a count of the number of connections that a node has. It may be expressed as Indegree (# of incoming connections) and Outdegre (# of outgoing connections).

12 ...Commonly Used Graph Algorithms... CLOSENESS CENTRALITY: Closeness considers the shortest paths between nodes and assigns a higher value to nodes that can be used to reach most other nodes most quickly. In the graph below, node A has the greatest centrality as all other nodes can be reached in one hop, whereas others require 1 hop to A or 2 hops to any other node. A

13 Commonly Used Graph Algorithms... CONNECTEDNESS: Check whether or not a set of nodes in a Graph are connected. All of the nodes in the graph below are connected, e.g. A to B, A to C via B etc. SHORTEST PATH: The path between two nodes that visits the fewest intermediate nodes. In the graph above, A->B->C->D is shorter than A->B->C->B->D (disallowing loops) NODE DEGREE: The degree of a node in a network is a count of the number of connections it has to other nodes. The degree distribution is the probability distribution of these degrees in the whole network. In the graph below, A and D have a node degree of 1. B andc have a node degree of 3.

14 ...Commonly Used Graph Algorithms... SHORTEST PATH: The path between two nodes that visits the fewest intermediate nodes. In the graph below, A->B->C->D is shorter than A->B->C->B->D (disallowing loops) AVERAGE PATH LENGTH: The average of all path lengths between all pairs of nodes in a graph. TRANSITIVE CLOSURE: The process of exploring a graph by traversing relationships until all nodes have been visited, but without revisiting nodes that are joined together in loops. In the graph above, A->B->C->D is a transitive closure.

15 ...Commonly Used Graph Algorithms... GRAPH DIAMETER (or SPAN): The greatest distance between any pair of nodes in a graph. It is computed by finding the shortest path between each pair of nodes. The maximum of these path thlengths is a measure of fthe diameter of fthe graph. The diameters of the two graphs below are 2 and 5.

16 ...Commonly Used Graph Algorithms... BETWEENESS CENTRALITY: A centrality measure of a node within a graph. Nodes that have a high probability of being visited on a randomly chosen short path between two randomly chosen nodes have a high betweeness In the graph below, node D has the highest betweeness centrality.

17 GRAPH DATABASES

18 Recognizing Graphs In Object Models... Tree Structures 1-to-Many Object Class A

19 ...Recognizing Graphs In Object Models... Tree Structures 1-to-Many Relationship Data Object Class A Object Class A

20 Recognizing Graphs In Object Models... Tree Structures 1-to-Many Relationship Data Object Class A Object Class A Graph (Network) Structures Many-to-Many Object Class A

21 Recognizing Graphs In Object Models... Tree Structures 1-to-Many Relationship Data Object Class A Object Class A Graph (Network) Structures Many-to-Many Relationship Data Object Class A Object Class A Copyright Objectivity, Inc. 2012

22 Why Do We Need Graph DBMSs?... Relational Database Think about the SQL query for finding all links between the two blue rows... Good luck! Table_A Table_B Table_C Table_D Table_E Table_F Table_G Relational databases aren t good at handling complex relationships!

23 ...Graph DBMSs Are Designed To Handle Relationships Relational Database Think about the SQL query for finding all links between the two blue rows... Good luck! Table_A Table_B Table_C Table_D Table_E Table_F Table_G Objectivity/DB or InfiniteGraph - The solution can be found with a few lines of code A3 G4

24 Graph Databases Data model: Node (Vertex) and Relationship (Edge) objects Directed May be a hypergraph h (edges with multiple l endpoints) Examples: InfiniteGraph, Neo4j, OrientDB, AllegroGraph, TitanDB and Dex VERTEX 2 N EDGE

25 Graph DBMSs Use A Very Simple Object Model Tree Structures 1-to-Many Relationship Data Object Class A Object Class A Graph (Network) Structures Many-to-Many Relationship Data GRAPH MODEL EDGE Object Class A Object Class A VERTEX Copyright Objectivity, Inc. 2012

26 Basic Capabilities Of Most Graph Databases... Rapid Graph Traversal Start

27 ...Basic Capabilities Of Most Graph Databases... Rapid Graph Traversal Inclusive or Exclusive Selection Start Start X X

28 ...Basic Capabilities Of Most Graph Databases Rapid Graph Traversal Inclusive or Exclusive Selection Start Start X X Find the Shortest or All Paths Between Objects Start Finish

29 CURRENT IMPLEMENTATIONS

30 Graph Databases Pre-2003

31 Graph Databases Post-2003 X

32 Graph Databases Compared [From OrientDB] Feature OrientDB Neo4j DEX InfiniteGraph License Open Source Open Source Commercial Apache and Commercial Query languages Not available, only via Via Java API Transaction support? ACID (plus lazy during bulk ingest) Protocols Embedded via Java API, remote as and Embedded via Java API and remote via REST? Embedded via Java API. Tinkerpop support. Replication Multi-Master Master-Slave No [No] Self loops Yes

33 Graph Databases Compared [UNSW] DATA STORAGE FEATURES

34 Graph Databases Compared [UNSW] OPERATION & MANIPULATION FEATURES

35 Graph Databases Compared [UNSW] GRAPH DATA STRUCTURES

36 Graph Databases Compared [UNSW] SCHEMA & INSTANCE REPRESENTATION

37 Graph Databases Compared [UNSW] QUERY FEATURES

38 Graph Databases Compared [UNSW] INTEGRITY CONSTRAINTS

39 Graph Databases Compared [UNSW] SUPPORT FOR ESSENTIAL GRAPH QUERIES

40 Graph Databases Pros and Cons Strengths: Extremely fast for connected data Scales out, typically Easy to query (navigation) Simple data model Weaknesses: May not support distribution or sharding Requires conceptual shift... a different way of thinking VERTEX 2 N EDGE

41 USE CASES

42 Example 1 - Market Analysis The 10 companies that control a majority of U.S. consumer goods brands

43 Example 2 - Demographics Used in social network analysis, marketing, medical research etc.

44 Example 3 - Seed To Consumer Tracking?

45 Example 4 - Ad Placement Networks Smartphone Ad placement - based on the the user s profile and location data captured by opt-in applications. The location data can be stored and distilled in a key-value and column store hybrid database, such as Cassandra The locations are matched with geospatial data to deduce user interests. As Ad placement orders arrive, an application built on a graph database such as InfiniteGraph, matches groups of users with Ads: Maximizes relevance for the user. Yields maximum value for the advertiser and the placer.

46 Example 5 - Healthcare Informatics Problem: Physicians need better electronic records for managing patient data on a global basis and match symptoms, causes, treatments and interdependencies to improve diagnoses and outcomes. Solution: Create a database capable of leveraging existing architecture using NOSQL tools such as Objectivity/DB and InfiniteGraph that can handle data capture, symptoms, diagnoses, treatments, reactions to medications, interactions and progress. Result: It works: Diagnosis is faster and more accurate The knowledge base tracks similar medical cases. Treatment success rates have improved.

47 Example 6 - Big Data Analytics

48 Example 7 Visual Analytics

49 Advice: The Repository Matters A Lot NEED RDBMS Key- Value Column Family Document Database ODBMS OLTP YES No Maybe No Maybe No Text Handling No No No YES Maybe No Graph Database Multimedia No Maybe No Maybe YES Maybe Engineering/ Scientific No No No No YES Maybe Business YES No Maybe No Maybe Maybe Intelligence Log Maybe No Maybe No YES Maybe Processing Connection Handling/ Analysis No No No No Maybe YES

50 More Advice: Languages and Tools Matter Too NEED Repository Language BI Tools Visual Analytics OLTP RDBMS SQL, Java YES Maybe Text Document Database Java, XML No Maybe Multimedia ODBMS Java, C++ No Maybe Eng/Science ODBMS C,C++, R Fortran Maybe YES Business RDBMS Java, SQL, R YES YES Intelligence Log NoSQL, C++, R, Processing ODBMS Java, SQL Connection Handling/ Analysis Graph Database Java, C++, SPARQL Maybe Maybe YES YES

51 A Polyglot Approach May Work Best LANGUAGE REPOSITORY PROBLEM ANALYTICS BI TOOLS GRAPH TOOLS VISUAL ANALYTICS

52 Hands On With A Graph Database We'll be using InfiniteGraph today You'll need a Java Development environment on your machine If you haven't downloaded InfiniteGraph already, please go to: [ We'll be covering a HelloGraph and a more complex sample program