Scaling-out Semantic Data Management and Processing

Transcription

1 Scaling-out Semantic Data Management and Processing Tomasz Wiktor Wlodarczyk, Norway

2 CIPSI Director: prof. Chunming Rong Areas of interest: Reasoning, analytics and simulation Distributed systems Dependability and security Focused on applied industrial research Scaling-out Semantic Data 2

3 Index Querying in Hadoop-based triple store Scaling-out NCBO Resource Index SEEDS, SCC, ERAC, A4Cloud Scaling-out Semantic Data 3

4 What is Hadoop? (CC) Introduction to Parallel Programming and MapReduce- Google Scaling-out Semantic Data 4

5 Querying in Hadoop - Motivation The goal was to: Demonstrate querying on triple-based data without any special storage structures on a distributed system Demonstrate querying for implicit data in a reallife scenario (dynamic datasets) Scaling-out Semantic Data 5

6 Querying in Hadoop - Data and Queries Lehigh University Benchmark 50, 100, 200, 500, 1000 and 6000 universities 13.6MB for 50 U and 2.9GB for 6000 U Queries used were 1 high selectivity, no reasoning required 2 complex interdependence pattern, simple reasoning 6 low selectivity, complex reasoning 14 low selectivity, no reasoning required LUBM focuses on RDF(S) and it was a problem in the context of OWL Run using on Amazon EC small and largenodes using AWS in Education Research Grant Scaling-out Semantic Data 6

7 Querying in Hadoop - Architecture Scaling-out Semantic Data 7

8 Querying in Hadoop Results (Reasoning) Scaling-out Semantic Data 8

9 Querying in Hadoop Results (Encoding) Plain text files Encoded files (using variable length integer) Encoded files (using byte array) Q1 Q2 Q14 Scaling-out Semantic Data 9

10 Index Querying in Hadoop-based triple store Scaling-out NCBO Resource Index SEEDS, SCC, ERAC, A4Cloud Scaling-out Semantic Data 10

11 NCBO Resource Index System for ontology based annotation and indexing of biomedical data Enable users to locate biomedical data resources related to particular concepts Semantic expansion is used to create search index Scaling-out Semantic Data 11

12 Scaling-out NCBO RI - Motivation Scaling-out Semantic Data 12

13 Scaling-out NCBO RI - Architecture Scaling-out Semantic Data 13

14 Scaling-out NCBO RI - Sets Used Scaling-out Semantic Data 14

15 Scaling-out NCBO RI - Joins Used Pig Hash Join Map + Reduce Pig Replicated Join Map only Pure MR Reduce Side Join Map + Reduce Pure MR Join Singleton Pattern Map + Reduce, can be Map only Scaling-out Semantic Data 15

16 Scaling-out NCBO RI - Results annotation_set_1 annotation_set_2 annotation_set_ Replicated Join in HDFS Hash Join in HDFS MR Reduce side join MR Joins With Singleton Scaling-out Semantic Data 16

17 Index Querying in Hadoop-based triple store Scaling-out NCBO Resource Index SEEDS, SCC, ERAC, A4Cloud Scaling-out Semantic Data 17

18 - Smart System to Support Safer Independent Living and Social Interaction for Elderly at Home Scaling-out Semantic Data 18

19 A4Cloud Accountability for Cloud Scaling-out Semantic Data 19

20 Other major projects SEEDS - Self learning Energy Efficient buildings and open Spaces SCC Computing -Strategic collaboration with China on super-computing based on Tianhe- 1A ERAC -Efficient and Robust Architecture for the Big Data Cloud Scaling-out Semantic Data 20

21 Summary Querying in Hadoop-based triple store Scaling-out NCBO Resource Index SEEDS, SCC, ERAC, A4Cloud Scaling-out Semantic Data 21

22 IEEE CloudCom th IEEE International Conference on Cloud Computing Technology and Science IEEE CloudCom 2012 The Grand Hotel, Taipei, Taiwan Dec 3 6, 2012 Keynotes from Shell, HP, VMWare, etc cloudcom.org Scaling-out Semantic Data 22