Not Relational Models For The Management of Large Amount of Astronomical Data. Bruno Martino (IASI/CNR), Memmo Federici (IAPS/INAF)

Size: px

Start display at page:

Download "Not Relational Models For The Management of Large Amount of Astronomical Data. Bruno Martino (IASI/CNR), Memmo Federici (IAPS/INAF)"

Noreen Gilmore
8 years ago
Views:

1 Not Relational Models For The Management of Large Amount of Astronomical Data Bruno Martino (IASI/CNR), Memmo Federici (IAPS/INAF)

2 What is a DBMS A Data Base Management System is a software infrastructure designed to operate on large data sets with the goal of optimizing their: storage access sharing protection

3 Why to use a DBMS? DBMS use command languages very flexible allowing among other things: imposition of consistency constraints creation of indexes to speed up the data access access to the logical data regardless of their physical representation interfacing with standard programming languages (C, C++, fortran, java )

imposition of consistency constraints creation of indexes to speed up the data

4 Types of DBMS The DBMS can be classified according to the following architectures: relational not relational The non-relational systems can still be divided into: reticular hierarchical NoSQL The most used architecture is the relational one.

non-relational systems can still be divided into: reticular

5 DBMS in HEP Well-known examples of the use of DBMS in high energy physics: SPIRES (SLAC Stanford Physics Information Retrieval System) LHC (CERN Large Hadron Collider data) OPERA (INFN Oscillation Project with Emulsion-tRacking Apparatus)!CHAOS (INFN Control system based on Highly Abstracted Operating Structure)

Hadron Collider data) OPERA (INFN Oscillation Project with Emulsion-tRacking

6 DBMS in astrophysics Well-known examples of the use of DBMS in astrophysics: Mars Express (INAF/INFOCOM MARSIS ground segment) MRO (INAF/INFOCOM SHARAD ground segment) ADS (SAO/NASA Digital Library for Physics and Astronomy) ANDROIDS (CFHT Andromeda Optical Infrared Disk Survey)

(INAF/INFOCOM SHARAD ground segment) ADS (SAO/NASA Digital Library

7 It looks easy FITS format files are widely used; their structure looks like a record in a database, isn t it? T Name Obs Mag Dia Obsvr V GOI C P2013J PAR03 The methodology to handle data in a DBMS within a user program follows a pattern very similar to the management of sequential files: mysql_real_connect (Conn, Host, User, DB, Psw, 0, null, 0); mysql_query (Conn, select * from Catalog where ID = 12");.. Result = mysql_store_result (Conn); Data = mysql_fetch_row (Result));... mysql_close (Conn); DataFil = (fopen ("Catalog", "r")); fscanf (DataFil, "%s", Data); fclose (DataFil);

7 PAR03 The methodology to handle data in a DBMS within a user program follows a pattern very similar to the management of sequential files:

8 Relational DBMS A relational DBMS is strictly based on tables (along the rows and columns of the matrix of data) that can be dependent on each other each column has a name and a data type (can not dinamically be changed) the standard command language is SQL some DBMS provides procedural extensions to the SQL base language (eg PL / SQL)

name and a data type (can not dinamically be changed) the standard command language

9 ACID In a relational DBMS a transaction (an individual, indivisible operation) works properly if it meets the following principles: Atomicity: the transaction must be either all or nothing, are not allowed partial executions Coherence: there must be no contradictions between the data stored in the DB Isolation: each transaction must be performed in isolated mode and independent from the others Durability: if a transaction requests a commit work, the changes will not be never lost

there must be no contradictions between the data stored in the DB Isolation: each transaction must be performed in

10 NoSQL DBMS They are called NoSQL because not necessarily based on the SQL language; the typologies are: key-value; association of the keys to their values that can be strings, lists, sets, etc document oriented; data are stored as documents. A document is a set of key/value pairs graph databases; data are stored as node set similar to the objects of the Object Oriented programming languages

document oriented; data are stored as documents.

11 CAP In 2000 Eric Brewer proposed for NoSQL DB the CAP Theorem: Consistency: all nodes in the distributed system must reflect the changes made Availability: the system is always able to respond to requests Partition Tolerance: if the communication is lost between two nodes the system continues to be reachable The theorem states that only 2 of the 3 requirements can be met simultaneously.

respond to requests Partition Tolerance: if the communication is lost between two nodes the

12 Design strategies A RDBMS design usually uses the following strategies: UML: Universal Modeling Language E/R: entity relationship model The schema-less nature of NoSQL databases means you can pretty much store your domain models as-is. You will still need to identify your distinct entities from your Key Value Objects. RDBMS NoSQL

of NoSQL databases means you can pretty much store your domain models as-is.

13 NoSQL data consistency NoSQL cannot necessarily give full ACID guarantees, usually eventual consistency is guaranteed (changes will be propagated to all the copies with finite delay) When processing scientific data the consistency constrain (C) may be more relaxed because few transactions are usually needed Access is usually carried out on shared data that are locally stored after their processing

processing scientific data the consistency constrain (C) may be more relaxed because few

14 MongoDB and others There is many NoSQL architectures, here we will analyze MongoDB, Cassandra and Hbase. MongoDB * HBase ** Cassandra ** Architecture master/slave master/slave p2p File system local HDFS local Parallel progr. mod. map/reduce map/reduce map/reduce Consistency strong strong weak (*) Already tested (**) See Future activities

MongoDB * HBase ** Cassandra ** Architecture master/slave master/slave p2p File

15 Horizontal scaling in MongoDB Data partitioning across multiple machines, is called horizontal scaling (scale-out) Sharding is MongoDB s approach to scaling out When a database s become too large, you need only to add a new machine, sharding automatically distributes data to the new servers

approach to scaling out When a database s become too large, you need

16 Data replication in MongoDB A replica set is a group of mongod instances hosting the same data set one mongod, the primary, receives all write operations all other instances, secondaries, apply operations from the primary so that they have the same data set

receives all write operations all other instances, secondaries,

17 MDBirs at IAPS/INAF 10 PC : i7 CPU 8 cores, 16GB RAM, 1 TB HD: 80 cores, 160 GB RAM 10 TB HD

18 MDBirs highlights The MDBirs system (MongoDB Retrieval Information System) has been designed to determine how much a NoSQL system is advantageous compared to RDBMS used for experiments as GAIA (ESA: Global Astrometric Interferometer for Astrophysics). Tests carried out using simulated data of GAIA on MDBirs showed that MongoDB guarantees: better query response times with increasing the amount of stored data scalability without intervention by the system manager auto-replication managed directly from MongoDB

Tests carried out using simulated data of GAIA on MDBirs showed that MongoDB guarantees: better query response times with

19 MongoDB performances By focusing attention on the performance of NoSQL DBMS is evident their greater speed compared to non-relational systems: VERY IMPORTANT! MongoDB becomes advantageous and competitive than MySQL only when the number of records in the data base is greater than There is no limit on the number of fields per record and the type of data, because there is no schema!

MongoDB becomes advantageous and competitive than MySQL only when the number of records in the

20 Future activities (IAPS/IASI/IAC) MongoDB is a good choice to obtain good safety and performances using local data sharding. Our goal is to achieve a full geographic data delocalization (CLOUD) in order to obtain: continuity of access to data (total disaster recovery) to the user community optimized average access time We have evaluated two possible scenarios: HBase: Hadoop Based NoSQL Data Base Cassandra: The Apache Cassandra database

Our goal is to achieve a full geographic data delocalization (CLOUD) in order to obtain: continuity of access

21 HBASE scenario HBase is a master-slave system including two types of machines: HMaster: access control HRegionserver: local data replication By using the Stargate plugin can be achieved local caches able to speed up response times by minimizing the network traffic to the master. Every region server keeps a copy of the data so the system ensures very high reliability.

22 Cassandra scenario It is a NoSQL solution initially developed by Facebook which has in the P2P architecture its focal point: it is decentralized: nodes in the cluster are identical. There is no single point of failure MongoDB * HBase ** Cassandra ** Architecture master/slave master/slave p2p it is fault-tolerant: data are automatically replicated on multiple nodes File system local HDFS local Parallel it is flexible: progr. mod. the throughput map/reduce of reading map/reduce or writing data map/reduce scales linearly with the addition of new machines (nodes), without Consistency strong strong weak downtime and without interruption of any application

23 Thanks to: Pietro Ubertini (IAPS-INAF Director) Paola Bertolazzi (IASI-CNR Director) Paola Marrese & GAIA staff (ASDC) Franco Giovannelli & staff (our great friends!)

Integrating Big Data into the Computing Curricula

Integrating Big Data into the Computing Curricula Yasin Silva, Suzanne Dietrich, Jason Reed, Lisa Tsosie Arizona State University http://www.public.asu.edu/~ynsilva/ibigdata/ 1 Overview Motivation Big