BIG DATA AGGREGATOR STASINOS KONSTANTOPOULOS NCSR DEMOKRITOS, GREECE. Big Data Europe

Transcription

1 BIG DATA AGGREGATOR STASINOS KONSTANTOPOULOS NCSR DEMOKRITOS, GREECE Big Data Europe

2 The Big Data Aggregator The Big Data Aggregator: o A general-purpose architecture for processing Big Data o An implementation of the core architecture v Integrating existing mature components o An ecosystem of tools around the core system v Driven by our use cases across all Horizon 2020 challenges

3 Conceptual basis Big Data Aggregator architecture builds upon the Lambda Architecture o Generic, scalable and fault-tolerant data processing architecture Batch layer o Time-consuming computations o Physically available data o Typically in large chunks

4 Conceptual basis Big Data Aggregator architecture builds upon the Lambda Architecture o Generic, scalable and fault-tolerant data processing architecture Speed layer o Computations expected to provide results in real time o Smaller amounts of data o Often streams

5 Conceptual basis Big Data Aggregator architecture builds upon the Lambda Architecture o Generic, scalable and fault-tolerant data processing architecture Data serving layer o Data input and data consumption o Results offered for consumption as views, predefined queries required by the application o Views combine batch and speed layer results to offer a unified view to the application

6 Big Data Aggregator Conceptual Architecture A Lambda Architecture for the Semantic Web o Generic, scalable and fault-tolerant data processing architecture o In the presence of semantic knowledge about the data o Maintaining metadata about provenance v Especially when pooling together multiple data sources v Including non-trivial data integration where substantial transformations are carried out

7 Big Data Aggregator Conceptual Architecture Background knowhow Bulk database Reporting API Background aggregator Bulk Data aggregator Real time aggregator Data Search index Dataset Meta data Data serving stores Data Serving API

8 Big Data Aggregator Conceptual Architecture System Admin User Interface Producer UI Background knowhow Bulk database Reporting API Background aggregator Bulk Data aggregator Real time aggregator Data Search index Dataset Meta data Data serving stores Data Serving API End user UI

9 Semantic Web aspects: Background Background knowledge o Integrating different pieces of background o Making it available to data processing Vertical links: o E.g., stream processor receives aggregated background

10 Semantic Web aspects: Provenance Provenance and other metadata o Metadata about data sources providing to this computation o Metadata travels down the processing pipeline without getting disassociated from the data it describes o Metadata is available as a data serving view Granularity o Per result tuple can become Big Data by itself o Per resultset can be less useful v Invalidates enormous processing for the slightest now-invalid input o Something inbetween or user configuration? v To be discussed

11 Semantic Web aspects: Data Semantic Web data o All component interfaces exchange RDF o Data serving API supports LD/SW formats v JSON, SPARQL & co o Besides ingesting RDF data and LD, run-time accessing SPARQL endpoints v At lease for the purposes of dynamically ingesting

12 Big Data Aggregator Conceptual Architecture Background knowhow Bulk database Reporting API Background aggregator Bulk Data aggregator RDF data procedding Real time aggregator Knowledge Knowledge Data Search index Dataset Meta data Data serving stores Data Serving API SPARQL JSON JSON-LD

13 From source code to deployed instance Source code management Github Packaging Debian Provisioning Docker Auto-deployment Chef Puppet Salt Ansible Swarm mesos

14 Starting Point Horton Works, o Integrated suite of Big Data processing tools LOD2 stack, o Tools for the Data Web o Ontologies o Automatically interlinking and fusing Web data o Provenance, privacy, security, quality o Searching, browsing, and authoring of Linked Data. SemaGrow, o Federated SPARQL querying o Data integration o Optimized query execution v Including over uncooperative endpoints o Provenance metadata GeoKnow, o Geospatial and LD integration o Data provenance o Adaptive geospatial exploration, authoring and curation

15 Thank you for your attention Questions?