Project Overview. Collabora'on Mee'ng with Op'mis, Sept. 2011, Rome

Transcription

1 Project Overview Collabora'on Mee'ng with Op'mis, Sept. 2011, Rome

2 Cloud-TM at a glance "#$%&'$()!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"#$%&!"'!()*+!!!!!!!!!!!!!!!!!!!,-./ !("*+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!&7"7#7"7!("*+!!!!!!!!!!!!!!!!!!!89:!;62!("$+! "$*+',%!,**$-.&#%*$)! )6/-/!8/46</=!"#$%&!"'!()*+! /0$#1*&)! >0/4!?@<9!ABCB!2/!D6E!ABCF! "$*2$#33')! >)GH"&*HABBIHJ!K!LMN95OP9!C7A! 40$%5'$!.&6*$3#1*&)! 3QRSTTUUU75-/@:2479@!

3 Cloud computing: the vision Cloud computing is at the peak of its hype

4 Cloud computing: (some) pitfalls Lack of programming models effec'vely hiding the issues of: concurrency distribu'on fault- tolerance elas'city Complexity Lack of effec've tools to automate elas'c scaling: manual monitoring expensive and error- prone automa'c resource provisioning tools s'll in their infancy Data consistency in dynamic systems is extremely challenging: no one size fits all solu'on

5 Key Question How to remove these roadblocks and materialize the Cloud vision? SIMPLIFYING THE DEVELOPMENT AND ADMINISTRATION OF CLOUD APPLICATIONS

6 Project Goals Develop an open- source middleware for the Cloud: 1. Providing a simple and intui've programming model: hide complexity of distribu'on, elas'city, fault- tolerance 2. Minimizing administra'on and monitoring costs: automate elas'c provisioning based on QoS/cost constraints 3. Minimize opera'onal costs via self- tuning adap'ng consistency mechanisms to maximize efficiency

7 The Cloud-TM Solution but first some background

8 From Transactional Memories Transac'onal Memories (TM): replace locks with atomic transac'ons in the programming language hide away synchroniza'on issues from the programmer avoid deadlocks, priority inversions, debugging nightmare simpler to reason about, verify, compose simplify development of parallel applica8ons

9 to Distributed Transactional Memories... Distributed Transac'onal Memories (DTM): extends TM abstrac'on over the boundaries of a single machine: enhance scalability ensure fault- tolerance minimize communica'on overhead via: specula'on batching consistency ac'ons at commit- 'me

10 to the Cloud-TM Programming Paradigm Elas'c scale- up and scale- down of the DTM plauorm: data distribu'on policies minimizing reconfigura'on overhead auto- scaling based on user defined QoS & cost constraints Transparent support for fault- tolerance via data replica'on: self- tuning of consistency protocols driven by workload changes Language level support for: transparent support of object- oriented domain model (incl. search) highly scalable abstrac'ons parallel transac'on nes'ng in distributed environments

11 Platform s architecture

12 Architecture Overview 31$"456*)01+2$.&) /+#+)01+2$.&) 0"?"$%>"E'&C$%&';&"CC)9;$/%!-$!"#$%$&'()*+%+,-.) J'#M8'-?$-N+8):8"B'9$/%!$ 2=O+8?$5&)*$ 4"NN+&$ #+"&8P$/%!$ 0)-?&)=<?+*$ 6Q+8<B'9$ R&"C+A'&S$ 7+8'9:;<&"=>+$0)-?&)=<?+*$ 7+8'9:;<&"=>+$#?'&";+$#D-?+C$ F27GH2/0$I$J'#$423!127$ F27GH2/0$/3/HKL67$ /0/%1/1!23$4/3/567$ 0"?"$%>"E'&C$ 2NBC)T+&$ 6>"-B8$#8">)9;$ 4"9";+&$ 8.$9'7'$%'%,):) ;<!)%-,$=+=$%)

13 Key Enabling Technologies Integra'on/extension of mainstream open source projects: leverage Red Hat s exper'se avoid reinven'ng the wheel focus on innova'on maximize project s visibility facilitate exploita'on of project s results provide a robust ini'al prototype

14 CLOUD-TM DATA PLATFORM

15 Reconfigurable Distributed STM 31$"456*)01+2$.&) /+#+)01+2$.&) 0"?"$%>"E'&C$%&';&"CC)9;$/%!-$!"#$%$&'()*+%+,-.) J'#M8'-?$-N+8):8"B'9$/%!$ 2=O+8?$5&)*$ 4"NN+&$ #+"&8P$/%!$ 0)-?&)=<?+*$ 6Q+8<B'9$ R&"C+A'&S$ 7+8'9:;<&"=>+$0)-?&)=<?+*$ 7+8'9:;<&"=>+$#?'&";+$#D-?+C$ F27GH2/0$I$J'#$423!127$ F27GH2/0$/3/HKL67$ /0/%1/1!23$4/3/567$ 0"?"$%>"E'&C$ 2NBC)T+&$ 6>"-B8$#8">)9;$ 4"9";+&$ 8.$9'7'$%'%,):) ;<!)%-,$=+=$%)

16 Reconfigurable Distributed STM (1) Infinispan was selected as reference plauorm: open source project led by JBoss in- memory transac'onal data grid key- value store API par'al and full replica'on

17 Some users of Infinispan Using Infinispan Today

18 Reconfigurable Distributed STM (2) Infinispan is being extended to support: addi'onal mechanisms for: data replica'on/distribu'on local concurrency control support for heterogeneous persistence storages online reconfigura'on mechanisms, e.g.: replica'on protocol switching dynamic data placement

19 Reconfigurable Storage System 31$"456*)01+2$.&) /+#+)01+2$.&) 0"?"$%>"E'&C$%&';&"CC)9;$/%!-$!"#$%$&'()*+%+,-.) J'#M8'-?$-N+8):8"B'9$/%!$ 2=O+8?$5&)*$ 4"NN+&$ #+"&8P$/%!$ 0)-?&)=<?+*$ 6Q+8<B'9$ R&"C+A'&S$ 7+8'9:;<&"=>+$0)-?&)=<?+*$ 7+8'9:;<&"=>+$#?'&";+$#D-?+C$ F27GH2/0$I$J'#$423!127$ F27GH2/0$/3/HKL67$ /0/%1/1!23$4/3/567$ 0"?"$%>"E'&C$ 2NBC)T+&$ 6>"-B8$#8">)9;$ 4"9";+&$ 8.$9'7'$%'%,):) ;<!)%-,$=+=$%)

20 Reconfigurable Storage System Plug- in based architecture suppor'ng heterogeneous persistent storages: local file systems cloud storage services (e.g. S3) Cassandra storage solu'ons developed by FP7 projects (TClouds)

21 Data Platform Programming API 31$"456*)01+2$.&) /+#+)01+2$.&) 0"?"$%>"E'&C$%&';&"CC)9;$/%!-$!"#$%$&'()*+%+,-.) J'#M8'-?$-N+8):8"B'9$/%!$ 2=O+8?$5&)*$ 4"NN+&$ #+"&8P$/%!$ 0)-?&)=<?+*$ 6Q+8<B'9$ R&"C+A'&S$ 7+8'9:;<&"=>+$0)-?&)=<?+*$ 7+8'9:;<&"=>+$#?'&";+$#D-?+C$ F27GH2/0$I$J'#$423!127$ F27GH2/0$/3/HKL67$ /0/%1/1!23$4/3/567$ 0"?"$%>"E'&C$ 2NBC)T+&$ 6>"-B8$#8">)9;$ 4"9";+&$ 8.$9'7'$%'%,):) ;<!)%-,$=+=$%)

22 Object Grid Mapper Two approaches are being explored to map a OO domain to the key/value model: JPA compliant API: Pro: widely adopted industry standard Con: constraints imposed by standard compliance Domain Modelling Language (DML) approach: Pro: increased flexibility Con: non- standard approach

23 Search API Support for JPA Query Language Integra'on with: Hibernate Search: OO query Apache Lucene: full text query 0$%5"5(5,-./)0%+1%'22'("3).-4!"#$%&'($)*+%$ *+%$ -B,B,-FGH 3+&?$%($% "&9$:$5)8)5$'%37$5 5$'%37)(+ 0$%5"5(5)"&(+!"#$%&'($) 6$'%37 5$'%37$5)8)"&9$:$5 5(+%$9)"& 9"5(%"#<($9) 4&!&"50'& 4&!&"50'& 4&!&"50'& 4&!&"50'& 4&!&"50'& 6$'%37)$&1"&$ '&9)'11%$1'("+& B#C$3(8D%"9)E'00$% Joins, aggrega'ons via Teiid: data virtualiza'on system allowing using data from mul'ple, heterogeneous data stores. ;$""9

24 Distributed Execution Framework Two main APIs: extension of Java 5 Concurrency APIs: scheduling and execu'on of tasks across the plauorm thread synchroniza'on mechanisms, e.g.: semaphores, queues, barriers concurrent, transac'onal data structures, e.g.: sets, hashmaps Map/reduce variant using data stored in the in- memory transac'onal data grid

25 AUTONOMIC MANAGER

26 Autonomic Manager 31$"456*)01+2$.&) /+#+)01+2$.&) 0"?"$%>"E'&C$%&';&"CC)9;$/%!-$!"#$%$&'()*+%+,-.) J'#M8'-?$-N+8):8"B'9$/%!$ 2=O+8?$5&)*$ 4"NN+&$ #+"&8P$/%!$ 0)-?&)=<?+*$ 6Q+8<B'9$ R&"C+A'&S$ 7+8'9:;<&"=>+$0)-?&)=<?+*$ 7+8'9:;<&"=>+$#?'&";+$#D-?+C$ F27GH2/0$I$J'#$423!127$ F27GH2/0$/3/HKL67$ /0/%1/1!23$4/3/567$ 0"?"$%>"E'&C$ 2NBC)T+&$ 6>"-B8$#8">)9;$ 4"9";+&$ 8.$9'7'$%'%,):) ;<!)%-,$=+=$%) 1st Annual Review, 4/7/2011, Bruxelles, Belgium 26

27 Self-tuning in babel! The Cloud- TM plauorm is an ecosystem of components using diverse technologies: JAVA, OCCI, OVF, Unix- dialects each with his own: interfaces parameters key performance indicators objec've func'ons Self- op'miza'on is already a challenge on his own: self- op'mizing a babel of components is impossible!

28 Generic Tunable Component Interface Abstrac'on layer hiding heterogeneity from Autonomic Manager XML encoding of metadata describing tuning op'ons and hints on rela'ons among op'ons: parameter types and ranges parameters to be considered KPIs parameters whose sejngs can affect KPI u'lity func'ons to use to self- tune each component

29 QoS/Cost specification API 31$"456*)01+2$.&) /+#+)01+2$.&) 0"?"$%>"E'&C$%&';&"CC)9;$/%!-$!"#$%$&'()*+%+,-.) J'#M8'-?$-N+8):8"B'9$/%!$ 2=O+8?$5&)*$ 4"NN+&$ #+"&8P$/%!$ 0)-?&)=<?+*$ 6Q+8<B'9$ R&"C+A'&S$ 7+8'9:;<&"=>+$0)-?&)=<?+*$ 7+8'9:;<&"=>+$#?'&";+$#D-?+C$ F27GH2/0$I$J'#$423!127$ F27GH2/0$/3/HKL67$ /0/%1/1!23$4/3/567$ 0"?"$%>"E'&C$ 2NBC)T+&$ 6>"-B8$#8">)9;$ 4"9";+&$ 8.$9'7'$%'%,):) ;<!)%-,$=+=$%)

30 QoS/Cost specification API Allow applica'ons for specifying agreements with the Cloud- TM plauorm concerning: QoS and cost constraints, e.g.: avg. transac'on execu'on 'me<1sec max opera'onal cost<100 /month applica'ons obliga'ons, e.g.: CPU 'me per transac'on<200msec transac'on conten'on probability<10e- 5 Reuse/extension of open source tools developed by recent FP7 projects: WS Agreement WSAG4J

31 Workload & QoS monitor 31$"456*)01+2$.&) /+#+)01+2$.&) 0"?"$%>"E'&C$%&';&"CC)9;$/%!-$!"#$%$&'()*+%+,-.) J'#M8'-?$-N+8):8"B'9$/%!$ 2=O+8?$5&)*$ 4"NN+&$ #+"&8P$/%!$ 0)-?&)=<?+*$ 6Q+8<B'9$ R&"C+A'&S$ 7+8'9:;<&"=>+$0)-?&)=<?+*$ 7+8'9:;<&"=>+$#?'&";+$#D-?+C$ F27GH2/0$I$J'#$423!127$ F27GH2/0$/3/HKL67$ /0/%1/1!23$4/3/567$ 0"?"$%>"E'&C$ 2NBC)T+&$ 6>"-B8$#8">)9;$ 4"9";+&$ 8.$9'7'$%'%,):) ;<!)%-,$=+=$%) 1st Annual Review, 4/7/2011, Bruxelles, Belgium 31

32 Workload & QoS monitor Efficient dissemina'on of monitoring data is essen'al in large scale systems Cloud- TM s monitoring framework is based on Lajce: monitoring framework developed in RESERVOIR differen'ated channels and specialized transport mechanisms: IP mul'cast, Pub/sub, etc extensions to enhance: portability across heterogeneous OSs/VMs efficiency in WAN/large datacenter sejngs

33 Workload Analyzer 31$"456*)01+2$.&) /+#+)01+2$.&) 0"?"$%>"E'&C$%&';&"CC)9;$/%!-$!"#$%$&'()*+%+,-.) J'#M8'-?$-N+8):8"B'9$/%!$ 2=O+8?$5&)*$ 4"NN+&$ #+"&8P$/%!$ 0)-?&)=<?+*$ 6Q+8<B'9$ R&"C+A'&S$ 7+8'9:;<&"=>+$0)-?&)=<?+*$ 7+8'9:;<&"=>+$#?'&";+$#D-?+C$ F27GH2/0$I$J'#$423!127$ F27GH2/0$/3/HKL67$ /0/%1/1!23$4/3/567$ 0"?"$%>"E'&C$ 2NBC)T+&$ 6>"-B8$#8">)9;$ 4"9";+&$ 8.$9'7'$%'%,):) ;<!)%-,$=+=$%)

34 Workload Analyzer.)&/-)"+$:*"-(6'&.)&/-)"+ 45"&"4#'&(6"#()*.)&/-)"+ 0)*(#)&!"#"$"%%&'%"#()* "*+$,(-#'&(*%.)&/-)"+$ "*+ &'7)8&4'7$87"%'$+'3"*+ 9&'+(4#()* :+"9#"#()* 0"*"%'& 1)2$3)*(#)&(*%$"*+$ $"-'&#$*)#(,(4"#()* 1)2$79'4(,(4"#()* "*+$ "-'&#$4)*,(%8&"#()*$

35 Adaptation Manager 31$"456*)01+2$.&) /+#+)01+2$.&) 0"?"$%>"E'&C$%&';&"CC)9;$/%!-$!"#$%$&'()*+%+,-.) J'#M8'-?$-N+8):8"B'9$/%!$ 2=O+8?$5&)*$ 4"NN+&$ #+"&8P$/%!$ 0)-?&)=<?+*$ 6Q+8<B'9$ R&"C+A'&S$ 7+8'9:;<&"=>+$0)-?&)=<?+*$ 7+8'9:;<&"=>+$#?'&";+$#D-?+C$ F27GH2/0$I$J'#$423!127$ F27GH2/0$/3/HKL67$ /0/%1/1!23$4/3/567$ 0"?"$%>"E'&C$ 2NBC)T+&$ 6>"-B8$#8">)9;$ 4"9";+&$ 8.$9'7'$%'%,):) ;<!)%-,$=+=$%)

36 Adaptation Manager

37 Key Project Phases Ini'al Pilot Applica'ons Demonstra'on Prototype of Distributed STM & Persistent Storage Final Pilot Applica'ons Final Clout- TM Prototype Prototype of the Autonomic manager Evalua'on YEAR1 YEAR2 YEAR3