Taming Big Data Variety with Semantic Graph Databases. Evren Sirin CTO Complexible

Transcription

1 Taming Big Data Variety with Semantic Graph Databases Evren Sirin CTO Complexible

2 About Complexible Semantic Tech leader since 2006 (née Clark & Parsia) software, consulting W3C leadership Offices in DC & Boston Launched Stardog 1.0 in 2012 Currently raising A Round

3

4 Big Data Vs Volume Velocity Variety Veracity Volatility Value

5 Data diversity is the real challenge Based on Paradigm4 survey of more than 100 data scientists

6 Data Variety Syntax: Formats Structure: Schemas Identity: Entities

7 In large and complex enterprises with lots of data, most analytic challenges can be reduced to data integration challenges.

8 Data integration approaches Integrated data Data warehouses Sweet spot Data lakes Integration effort

9 What is a Unified Data Model? Global coherent view over heterogenous data flexible and composable at the right level of abstraction enabling automated processing and analysis

10 Data models Tables Trees Graphs

11 Data models Tables Trees Graphs

12 Data models Tables Trees Graphs

13 Data models Tables Trees Graphs

14 Graphs are everywhere Knowledge Graph Open Graph Linked Open Data

15 Why graphs? Generic data representation model Utilize connectedness of the data Flexible and extensible Easy to compose and connect Increasing number of graph database offerings

16 not Why graphs? Generic data representation model Utilize connectedness of the data Flexible and extensible Easy to compose and connect Increasing number of graph database offerings

17 Semantic graphs = RDF graphs Meaning is defined in an explicit and machine-processable way

18 Abstract Graph

19 RDF Graph

20 RDF Serialization 01 BASE < 02 PREFIX foaf: < 03 PREFIX xsd: < 04 PREFIX schema: < 05 PREFIX dcterms: < 06 PREFIX wd: < <bob#me> 09 a foaf:person ; 10 foaf:knows <alice#me> ; 11 schema:birthdate " "^^xsd:date ; 12 foaf:topic_interest wd:q wd:q dcterms:title "Mona Lisa" ; 16 dcterms:creator < < 19 dcterms:subject wd:q SPARQL Query PREFIX foaf: < PREFIX schema: < PREFIX dcterms: < PREFIX dbpedia: < SELECT?person?title WHERE {?person a foaf:person ; schema:birthdate?birthdate ; foaf:topic_interest?interest.?interest dcterms:title?title ; dcterms:creator dbpedia:leonardo_da_vinci. FILTER (?birthdate < " "^^xsd:date ) }

21 Schema - Ontology rdfs:subclassof Agent rdfs:subclassof worksfor owl:inverseof hasemployee Person Organization rdfs:range rdf:type rdf:type rdf:type rdf:type Bob worksfor ACME hasemployee

22 Tables to graphs (R2RML)

23 RDF Models Interoperable - No vendor lock-in Actionable - Run queries against it Expressive - Multiple views over same data Reusable - By different apps in other domains

24 Model-based data integration

25 Stardog: Semantic Graph Database The leading RDF database Pure Java: any JVM language, full REST bindings Client-server, embedded, middleware modes Rich feature set ACID Transactions, High Availability, Hot backup/restore, JMX server monitoring, Access & Audit logging, RBAC security model, LDAP integration, SPARQL 1.1 queries, OWL 2 Reasoning, Proof trees, Integrity constraints, Full-text search, Geospatial support, Virtual graphs, Provenance support Supports property graphs (Tinkerpop)

26 Single-node Scalability Scale up to 50B triples on modest hardware 32 cores, 256 GB RAM, 2 x 7200RPM HDDs, < $10K cost Load rates up to 500k triples/second That s 100M triples in 3 min, 1B in 30 min, and 20B in 20 hours Best-of-breed query answering performance Query 100M triples with a throughput of 3M+ queries/hour, 1B at 500k queries/hour, and 10B at 20k queries/hour (BSBM, 64 clients)

27 Stardog for Big Data HDFS-backed storage Horizontal partitioning of data Advanced query planner and optimization Parallel query execution with async messaging Coming in version 5 (2016)

28 Big Data computations Integration with Apache Spark Run Spark jobs over integrated view of the data PageRank, machine learning algos,... Different ways to expose RDF data in Spark RDD[Triple] RDD[SPARQLResults]

29 Thanks!