Safely Managing Data Variety in Big Data So2ware Development

Transcription

1 Safely Managing Data Variety in Big Data So2ware Development Thomas Cerqueus (Université de Lyon, INSA Lyon, LIRIS) Eduardo Cunha de Almeida (Federal University of Paraná) Stefanie Scherzinger (OTH Regensburg) 1st InternaRonal Workshop on BIG Data So2ware Engineering, 23 May 2015

2 PaaS over NoSQL Databases No explicit DB schema EnRRes are defined at the applicaron level Class è EnRty A]ribute è Field Developers are in charge of defining and maintaining the DB

3 Object Mappers

4 Object Mappers

5 development IDE commit code repository Development Environment ProducRon Environment 5

6 deploy development IDE commit PaaS DaaS code repository Development Environment ProducRon Environment 6

7 deploy development IDE commit PaaS DaaS code repository Development Environment ProducRon Environment 7

8 development IDE commit PaaS code repository Development Environment ProducRon Environment 8

9 development IDE commit PaaS code repository Development Environment ProducRon Environment 9

10 development IDE commit PaaS code repository Development Environment ProducRon Environment 10

11 Checking Rules (1/2) C W type att; C R C W type att; ok C W ` type att; C R C W ` type att; ok C R C W ` type att; ok C W type 1 att; C R C W ` type 2 att; ok att not declared in C W C W ` type att; C R C W type att; ok C W ` type att; C R C W type att; ok C W ` type att; C R C W type att; ok (1) (2) (3) Fig. 4: Selected rules for (1, 2) keeping an attribute, lazily (3, 4) adding, (5) overwriting, and (6, 7) explicitly removing an attribute. (4) (5) (6) (7) In the example above, different type. If the type checking rules would not capture this case as well, chains of object mapper in Section IV-B. IV. RELAXA We next discuss pragm A. Allowable Type Prom Our examples so far h class member attributes, easily generalize to comp Example 6. We conside different versions of the Player c Player d The Objectify annotation

12 Checking Rules (2/2) C W ` type att 1 ; C R C W 2 ") type att 1 ; ok C W ` type att 2 ; C R C W 2 ") type att 1 ; ok C W type att 1 C W ` type att 2 ; C R C W 2 ") type att 1 ; ok C W ` type att 1 C W type att 2 ; C R C W 2 ") type att 1 ; ok (att 2 not declared in C W ) (8) (att 1 not declared in C W ) (9) (@annot (10) (@annot (11) (att 1 and att 2 not declared in C W ) (12) C R C W 2 ") type att 1 ; ok C W 1 type att 1 2 are any of (13) C R C W 2 type att 1 @IgnoreLoad Fig. 5: Selected rules for annotation (8-12) and for ignoring attributes (13). c) Conversion Returning a Corrupted Value: Surpris-, there are cases where Objectify allows a conversion,

13 Example C W C R C W type att; C R C W type att; ok C W ` type att; C R C W ` type att; ok C R C W ` type att; ok att not declared in C W (1) (2) (3) In the example a different type. If th checking rules wou capture this case as chains of object m in Section IV-B. IV. RE

14 Example C W C R C W type att; C R C W type att; ok C W ` type att; C R C W ` type att; ok C R C W ` type att; ok att not declared in C W C W type 1 att; C R C W ` type att; ok (1) (2) (3) (4) In the exam different type checking rule capture this c chains of obj in Section IV IV We next di

15 Example C W C R C W type att; C R C W type att; ok C W ` type att; C R C W ` type att; ok C R C W ` type att; ok C W type 1 att; C R C W ` type 2 att; ok att not declared in C W C W ` type att; C R C W type att; ok (1) (2) (3) (4) (5) In the example abov different type. If the ty checking rules would n capture this case as we chains of object mapp in Section IV-B. IV. RELAX We next discuss pra A. Allowable Type Pro Our examples so fa class member attribute

16 Example C W C R

17 an attribute, lazily (3, 4) y removing an attribute. ts rules concerning the mbiguous mappings in Figure 5 concerns the writing or reading. Igeely without risk. ules sitive in the following Y, and Z of an object ecks as valid w.r.t. X, it does not necessarily.t. X. It turns out that ctual problems, as the f versions X, Y, and on, with the following The Objectify stores structured data within a single entity in a way so that it remains queryable. We assume that A and B are different Java class names. Type Checking Extension By our type checking approach, Player d does not check as In the e valid w.r.t Player C W c, since type theatt; types for attribute description different t (1) differ. Yet C R there C W are cases type where att; this okrule is overly restrictive. checking Rule (14) below relaxes rule (2). We use the notation tbt 0 to capture th C W ` type att; express that type t is compatible with t 0. This means (2) that any chains of C instance R C of t must W ` type att; ok be loadable as an instance of t 0 without in Section data loss. The trivial case is that t = t 0 att not, declared yet we can also (3) allow for C R Short C W B` Integer, type att; since okall instances C W of Short can be converted to Integers. We nex C W type 1 att; type att; (4) C R C W ` type C R C W ` type 0 (type B type 0 2 att; ok ) (14) A. Allowa att; ok C W ` type att; Our ex We look at type compatibility in more detail. (5) C R C W type att; ok class mem 1) Conversion of Primitive Types: For the remainder of this easily gen section, we focus on C W primitive ` type att; types 1 (i.e., Void, (6) Boolean, Byte, Short, C R CInteger, W Long, Float, type Double, att; Character, ok String Example and Object), and C we discuss the risks when primitive type different v W ` type att; attributes are loaded from a datastore. (7) C R C W type att; ok P

18 Type Checking Extension PrimiRve Types Boolean Byte Short Integer Long Float Double String Object Boolean Byte Short Integer Long Float Double String Boolean Long Long Long Long Double Double String Excep;on at run;me Precision loss OK Unexpected type conversion

19 Type Checking Extension Complex Types X Y A a B b C c X A a B b C c Y

20 Chains of DeclaraRons

21 Conclusion ControVol: schema evoluron checking framework for NoSQL databases Takes acron prior to applicaron release Integrated in the IDE (Eclipse) Warns developers against perilous evolurons: addiron/deleron/renaming of a]ributes, re- typing Suggests quick fixes

22 Conclusion

23 PublicaRons T. Cerqueus, E. Cunha de Almeida, S. Scherzinger. ControVol: Let yesterday's data catch up with today's applica;on code. In 24th Interna3onal World Wide Web Conference (WWW 2015) S. Scherzinger, T. Cerqueus, E. Cunha de Almeida. ControVol: A Framework for Controlled Schema Evolu;on in NoSQL Applica;on Development. In 31st Interna3onal Conference on Data Engineering (ICDE 2015) T. Cerqueus, E. Cunha de Almeida, S. Scherzinger. Safely Managing Data Variety in Big Data SoQware Development. In 1st Interna3onal Workshop on Big Data SoEware Engineering (BIGDSE'15)

24 Safely Managing Data Variety in Big Data So2ware Development Thomas Cerqueus (Université de Lyon, INSA Lyon, LIRIS) Eduardo Cunha de Almeida (Federal University of Paraná) Stefanie Scherzinger (OTH Regensburg) 1st InternaRonal Workshop on BIG Data So2ware Engineering, 23 May 2015