DataStorm 2013 Workshop on Large-Scale Data Management

Transcription

1 Domain Specific Languages for Large- Scale-Data Applications DataStorm 203 Workshop on Large-Scale Data Management 6/7/203 Alberto Rodrigues da Silva (on behalf of the Information Systems Group, INESC-ID) Agenda Big Data DataStorm Scope Models and Modeling Languages DSL for Big Data Model-Driven Engineering Approach Discussion 2

2 Big Data Properties Volume Terabytes Transactions Tables, Files Variety Structured semistructured Unstructure Velocity Batch Near time Real time Streams 3 Big Data Data Sources Transactions Enterprise Data Social Media Public Data Sensor Data 4 2

3 Big Data Hadoop Ecosystem Storage: HDFS Distributed file system Other tools/technologies Avro (serialization system) Pig (data flow language) Hive (distributed data warehouse) ZooKeeper (coordination service) Analysis: MapReduce Distributed data processing and execution environment 5 Big Data Concepts 6 3

4 DataStorm Scope Horizontal Tasks H: Data Acquisition and Information Extraction H2: Data representation, querying and validation H3: Knowledge discovery from heterogeneous data H4: Information Lifecycle Management H5: Contexts and Semantics of Information H6: Domain Specific Languages for Large-scale- Data Applications Vertical Tasks General Scope V: Biomedical Data Resources and Data Processing Infrastructure V2: Environmental Data Resources and Data Processing Infrastructure V3: Societal Data Resources and Data Processing Infrastructure V4: Cultural Data Resources and Data Processing Infrastructure 7 DataStorm Scope H6 Task s Objectives Research and evaluate the design and use of DSLs (domain specific modeling languages) to improve and normalize the design of this class of applications. 8 4

5 DataStorm Scope DSL Examples Typical two-level network architecture for a Hadoop cluster 9 DataStorm Scope DSL Examples MapReduce 0 5

6 DataStorm Scope DSL Examples In Pig Latin Background Objectives Why do we need DSLs and modeling languages? To better Share the knowledge of the application domain and facilitate the communication among the involved stakeholders Plan the projects/tasks involved And, eventually, to promote effective and efficient ways to design and to develop Big Data apps through modeldriven approaches 2 6

7 Core Concepts Models and Metamodels Represents > +sus System +model * * Model * ElementOf > Modeling Language InstanceOf > Metamodel * Represents > 3 Core Concepts Modeling Languages Modeling Language Abstract Sintax «synonym» 0.. Metamodel Static Semantics..* Concrete Sintax «synonym» Notation * Semantics * Pragmatic 4 7

8 MDE/MDD Considers Models as first class citizen artefacts and allow the production or automatic execution of software systems based on these models, through techniques such as: - metamodeling - model transformations - code generation - models interpretation 5 MDE/MDD Some popular initiatives - MDA (Model Driven Architecture), by OMG - Software Factories, by Microsoft Initiatives and Tools 6 8

9 MDE/MDD In one picture! Plataform Transformation +source Model +target +executable model Non-Generated Artifact Model-Plataform Transformation Model-Model Transformation Software Product Generated Artifact +target 7 Our recent and related research Some Thesis T UML is not enough and appropriate for defining UIs so we proposed the XIS UML Profile - Carlos Martins, Modelação de Interfaces Gráficas no âmbito do ProjectIT (MEI/UMA), MSc Thesis, Univ. Madeira, Nov Raul Queiroga, LEIC, IST/UTL, Arquitecturas de Software no projecto XIS, Relatório de TFC, LEIC, IST, Outubro de Gonçalo Lemos & Tiago Matias, LEIC, IST/UTL, XIS - extended Interactive Systems, Relatório de TFC, LEIC, IST, Julho de

10 Our recent and related research Some Thesis T2 In many situations we need to provide multiple related languages to better support different stakeholders - We validate this in the context of CMS-based Web Apps João de Sousa Saraiva, Development of CMS-based Web Applications with a Multi- Language Model-Driven Approach, PhD Thesis, IST/UTL Jan/203 9 The Proposal (in the Scope of DataStorm) ) Surveying existing DSLs for Big Data apps 2) Design a Visual DSL for Big Data Apps As a UML profile, to take benefit from UML Tools, e.g. Enterprise Architect Multiple Views, Multiple Concerns 3) Support and get feedback from other Vertical Tasks (i.e., Biomedical, Environmental, Societal, Cultural domains) 20 0

11 BigData DSL Concepts Meta-concepts - Cluster, Node, JobTracker, TaskTracker - Job, Task, Mapper, Reducer - Data Resource, File, Directory, File Pattern - Data Entity, Record, Key-value Tuple - Stream, Split - 2 BigData DSL Multiple views Business-level views - Domain and Business entities view - Analysis view Technological-level views - Architecture/Deployment view - Data view - Job View 22

12 Thank you! Alberto Rodrigues da Silva IST / INESC-ID Department of Computer Science and Engineering alberto.silva@inesc-id.pt future 24 2