Recap& What%is% big%data % Introduction%to%e4science% Reflections%of% big%data %in%health% Guest%lecture:%% DB%for%big%data%analytics%

Transcription

1 Recap& What%is% big%data % Introduction%to%e4science% Reflections%of% big%data %in%health% Guest%lecture:%% DB%for%big%data%analytics% 1$

2 Recap& Data%infrastructure% Sources,%capture,%representation,%access% Analytics% Many%considerations%4>%how%to%do%it%in%practice?% % 2$

3 Conceptual&architecture& Raghupathi and Raghupathi Health Information Science and Systems 2014, 2:3 3$

4 Methodology& Raghupathi and Raghupathi Health Information Science and Systems 2014, 2:3 4$

5 Example:&World9Wide&Pleksys& Data%infrastructure%to%study%long4term%effect%of%cancer%in%children% 5 min Patients& Healthcare&& Registration& Research&& 5$

6 Large-scale Infrastructure 6$

7 Large&scale&processing& MIK%2.1%Databases%and%Networksystems% Big%Data%week% % Silvia%Delgado%Olabarriaga,%KEBB%

8 Infrastructure&capacity& Computing%time% Memory%size% Data%storage%(input/output)% Data%Transport% Capacity%has%large%influence%on%software%construction% Scalability%is%the%ability%of%a%system,%network,%or%process%to% handle%a%growing%amount%of%work%in%a%capable%manner%or% its%ability%to%be%enlarged%to%accommodate%that%growth.% 8$ % %

9 Data&infrastructure:&Storage& Storage%Technologies% Disk% Tape% Optical% Composite%devices% Storage%Parameters% Block%size% Random%vs.%sequential%access% Bandwidth% Media%integrity,%data%integrity% Price% See Shoshani and Rotem(eds.) Scientific Data Management: Challenges, Technology and Deployment, chapter 1. CRC Press, $

10 Computing&infrastructures& Parameters%%% Speed% Memory% Throughput% Programming% Types% High4performance& computing%%(hpc)% High4throughput% computing%(htc)% Virtual%computing% Cloud% 10$

11 Distributed&Computing& Field%of%computer%science%that%studies%distributed%systems% Distributed%system%=%software%system%in%which%components% located%on%networked%computers%communicate%and% coordinate%their%actions%by%passing%messages.% components%interact%for%common%goal% concurrency%of%components% lack%of%a%global%clock% independent%failure%of%components% Examples:%grid,%service%oriented%architecture%(SOA)% 11$

12 Parallel&computing& Form%of%computation%in%which%many%calculations%are%carried% out%simultaneously% Principle%=%large%problems%can%often%be%divided%into%smaller% ones,%which%are%then%solved%concurrently%("in%parallel").%% Forms%of%parallel%computing:%% bit4level%(word%size)% instruction4level%(pipeline)% data4level% task4level% 12$

13 Distributed&vs.&parallel&computing& The%terms%"concurrent%computing",%"parallel%computing",% and%"distributed%computing"%have%a%lot%of%overlap% distributed parallel Multicore 13$

14 Types&of&Computing&Infrastructures& Types% High4performance%computing%%(HPC)% High4throughput%computing%(HTC,%Grid)% Virtual%computing%(Cloud)% % Parameters%%% Speed% Memory% Throughput% Programming% 14$

15 Supercomputer Cluster GPU High&Performance&Computing& 15$

16 Supercomputer& computer%at%the%frontline%of%current%processing%capacity,% particularly%speed%of%calculation% memory,%processors%% optimized%architecture% TOP500% 16$

17 17$

18 18$

19 Cluster& Set%of%loosely%connected%computers%that%work%together%so% that%in%many%respects%they%can%be%viewed%as%a%single%system% Homogeneous%vs.%heterogeneous% Fast%network% Shared%data%space% Head%node,%worker%nodes% Batch%(queue)% % 19$

20 20$

21 21$

22 Graphics&Processing&Unit&9&GPU& Massively%parallel% Origin%(80 s)% Specialized%unit%to%accelerate%drawing%of%graphics%(lines,%circles,% characters)%on%display%memory% From%2006:%% Stream%processing%(single%instruction,%multiple%data)% CUDA% OpenCL% 22$

23 SOAP3 aligner GSNP (SNP detection) GAMA (high resolution genotyping tool) gpu-based-server-farm-slashes-analysis-time-for-dna-sequencing-data-batches-from-days-to-hours 23$

24 Recap& HPC%% Supercomputer% Cluster% GPU% Programming?% 24$

25 HPC&Programming&(1)& Parallelism% Task% Data% task data Coordination% Coupled%(tight,%loose)% Independent%( embarrassingly %parallel)% 25$

26 HPC&Programming&(2)& Middleware% Portable%batch%system%(pbs)% Sun%grid%engine%(SGE)% Hadoop% Programming%interfaces%(APIs)% Message%passing%interface%(MPI,%Open%MPI)% Open%Multi4Processing%(OpenMP)% % Languages% CUDA% OpenCL% % 26$

27 Recap& HPC% Supercomputer% Cluster% GPU% Programming% Middleware% APIs%and%libraries% Programming%Languages% In%NL%(SURFsara)% Cartesius% LISA% 27$

28 GRID% High&Throughput&Computing& 28$

29 Grid& a%system%that%uses%open,%general4purpose%protocols%to% federate%distributed%resources % resources:%computing,%data,%storage,%services,%software,%equipment,% expertise% Jan%Foster,% What%is%the%grid?%A%three%point%check%list,%2002%% Key:%FEDERATION% organization%or%group%within%which%smaller%divisions%have%some% degree%of%internal%autonomy % 29

30 History& Term%originated%early%1990s%% computer%power%as%easy%to%access%as%in%an%electric%power%grid% Ian%Foster's%and%Carl%Kesselman,%%"The%Grid:%Blueprint%for%a%new% computing%infrastructure,%1999% Around%1997:%CPU%scavenging%and%volunteer%computing% harness%the%power%of%networked%pcs%worldwide,%in%order%to%solve% CPU4intensive%research%problems% 30$

31 Basic&concept& User interface (UI) Resource Broker Worker Nodes 31$

32 Virtual&Organizations&(VO)& SECURITY& Site A Site B 32$

33 Grid&Security& Authentication,%Authorization,%Accounting%(AAA)% Standard%certificate%infrastructure% public%key%asymmetric%encryption,%x.509%certificates% User%certificate% Identity% VO4membership% Short4lived%grid&proxy&certificates% Single%sign4on% ftp://ftp.globus.org/pub/globus/papers/security.pdf 33$

34 How&to&program& a&grid?& Grid$ proxy$ program Step$2$ Step$3$ grid 34$

35 How&to&program& a&grid?& Step$2$ Step$3$ Stage-in grid 35$

36 How&to&program& a&grid?& Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Step$2$ Step$3$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Stage-in Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ grid 36$

37 How&to&program& a&grid?& Step$2$ Step$3$ Step$2$ Step$3$ Data2$ Step$2$ Step$3$ Data3$ Step$2$ Step$3$ Data4$ Step$2$ Step$3$ Data5$ Step$2$ Step$3$ Data6$ Step$2$ Step$3$ Data7$ Step$2$ Step$3$ Data8$ Step$2$ Step$3$ Data9$ grid Grid jobs Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ 37$

38 How&to&program& a&grid?& Step$2$ Step$3$ Step$2$ Step$3$ Data2$ Step$2$ Step$3$ Data3$ Step$2$ Step$3$ Data4$ Step$2$ Step$3$ Data5$ Step$2$ Step$3$ Data6$ Step$2$ Step$3$ Data7$ Step$2$ Step$3$ Data8$ Step$2$ Step$3$ Data9$ grid Grid jobs Generate results Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ Grid$ proxy$ 38$ Result$ Result$ Result$ Result$ Result$ Result$ Result$ Result$ Result$

39 How&to&program& a&grid?& Results1$ Stage-out grid 39$

40 How&to&program& a&grid?& Grid$ proxy$ program Step$2$ Step$3$ Results1$ grid 40$

41 Grid&Middleware& Middleware-=%connects%software%components%or%people%and% their%applications.% Grid-middleware:% Application- submits-a-job -to%execute%a%computation- resource-broker %distributes%the%job%to%the%node%that%seems%to%be% the%best%suitable%for%this%job.% Files%(data,%results)%are%transferred%to/from%the%nodes%(data%staging)% Can%be%seen%as%the%operating-system-of%the%grid.% 41$

42 Example:&BOINC& Berkeley%Open% Infrastructure%for% Network%Computing% Voluntary%grid% 42$

43 Example:&gLite& produced%by%the%egee%project%% Until%May%2014%4%EMI%project% Used%by%EGI% % 43$

44 European&Grid&Initiative& % More%info:% % 44$

45 45$

46 Recap& HTC% Grid%computing% Jobs% Federation% Security%(AAA)% Various%middleware% glite% In%NL%(SURFsara)% Life%Science%Grid% EGI% 46$

47 CLOUD& Virtual&Computing& 47$

48 Cloud&Computing& Internet4based%computing% shared%resources,%software%and%information%are%provided%to% computers%and%other%devices%on4demand,%similar%to%a%public%utility% such%as%the%electricity%grid.% Refers%to%both%% the%applications%delivered%as%services%over%the%internet%and% the%hardware%and%systems%software%in%the%data%centers%that%provide% those%services.% Virtualization% %software%systems%that%act%as%if%they%were%hardware%systems% $

49 Vision& 49$

50 Cloud&Computing& % Types% Public%cloud% Private%cloud% Utility%computing:%service%being%sold% Novelties% appearance%of%infinite%computing%resources%available%on%demand% ability%to%pay%for%use%% Michael Armbrust et al. A View of Cloud Computing. Communications of the ACM, Vol. 53 No. 4, Pages $

51 .&XXX&As&a&Service& Infrastructure%as%a%service%(IaaS)% Platform%as%a%service%(PaaS)% Software%as%a%service%(SaaS)% Storage%as%a%service%(STaaS)% Network%as%a%service%(NaaS)% Security%as%a%service%(SECaaS)% Data%as%a%service%(DaaS)% Database%as%a%service%(DBaaS)% Test%environment%as%a%service%(TEaaS)% API%as%a%service%(APIaaS)% Backend%as%a%service%(BaaS)% % 51$

52 IAAS:&Infrastructure&as&a&Service& Looks%like%physical%hardware% Amazon%Elastic%Compute%Cloud%(Amazon%EC2):%web%service%that% provides%resizable%compute%capacity%in%the%cloud.% Dutch%HPC%Cloud% 52$

53 How&to&program& a&cloud&(iaas)?& program Step$2$ Step$3$ Data$ cloud 53$

54 How&to&program& a&cloud&(iaas)?& Virtual Machine Virtual Machine Virtual Machine Virtual Machine Step$2$ Step$3$ Step$2$ Step$3$ Step$2$ Step$3$ Step$2$ Step$3$ Data$ Data$ Data$ Data$ cloud 54$

55 How&to&program& a&cloud&(iaas)?& Virtual Machine Virtual Machine Step$2$ Step$3$ Step$2$ Step$3$ Virtual Machine Virtual Machine Step$2$ Step$3$ Step$2$ Step$3$ cloud 55$

56 How&to&program& a&cloud&(iaas)?& Virtual Machine Virtual Machine Step$2$ Step$3$ Step$2$ Step$3$ Virtual Machine Virtual Machine Step$2$ Step$3$ Step$2$ Step$3$ cloud 56$

57 How&to&program& a&cloud&(iaas)?& Virtual Machine Virtual Machine Virtual Machine Virtual Machine Step$2$ Step$3$ Step$2$ Step$3$ Step$2$ Step$3$ Step$2$ Step$3$ Data$ Data$ Data$ Data$ cloud Results' 57$

58 How&to&program& a&cloud&(iaas)?& Data$ program Step$2$ Step$3$ Results' Results' Results' Results' cloud 58$

59 Cloud&Middleware& Basics% Images% Virtualization% Management% Basic%layer:%Virtualization%service% VMWare% Virtual%Box% KVM% Management%layer:%% Eucalyptus% OpenNebula% OpenStack%% Open%Cloud%Computing%Interface% 59$

60 60$

61 Putting&it&all&together:&& computing&infrastructures& e-infrastructure cloud computing high throughput computing high performance computing grid parallel programming 61$

62 Putting&it&all&together:&& computing&infrastructures& e-infrastructure pay-per-use on demand cloud computing "as service" IAAS (infrastructure) SAAS (software) VMWare cluster GPU supercomputer Occi Eucalyptus middleware virtual machine VirtualBox high performance computing OpenStack MPI OpenMP API parallel programming languages CUDA OpenCL task parallelism data parallelism high throughput computing grid proxy private key public key job data staging grid certificate grid middleware production grid BOINC glite Condor EGI Dutch e-science Grid virtual organization 62$

63 63$

64 Take&home&message& Large%scale%Computing%infrastructures% Supercomputer,%Cluster,%GPU% Grid,%Cloud% Parallelism% Data,%tasks% Interfaces% Interactive%(command4line)% Middleware% Programming%languages/frameworks% 64$

65 Questions?&