ERAC. Efficient and Robust Architecture for the big data Cloud. Tor Skeie

Transcription

1 ERAC Efficient and Rbust Architecture fr the big data Clud Tr Skeie

2 Clud cmputing challenges Results f a survey amng enterprises, cnducted by IDC Surce: IDC Enterprise Panel 2009

3 Perfrmance requires pwer The IT industry is respnsible fr 2,0% f the CO 2 emissin wrldwide, the same amunt as the aircraft industry. Cnsumes 2% f the glbal pwer prductin and that is increasing significantly, assumed t be 4% in 2015 PUE (Pwer Usage Effectiveness) is a measure f hw efficiently a cmputer data center uses its pwer; specifically, hw much f the pwer is actually used by the cmputing equipment in cntrast t cling and ther verheadnd ther verhead PUE fr a datacenter is typically 2,0 Cmputatinal pwer Cling If all the data centers in the wrld had a PUE arund 1.6, the Kyt-agreement wuld be met 50 % 50 % A green data center shuld have a PUE f nt mre than 1.2

4 Clud datacenters have pr utilizatin efficiency tday Pwer efficiency f 19 datacenters CPU utilizatin f 5000 randm Ggle servers ver a perid f six mnths Surce: Cmputer Architecture: A Quantitative Apprach. By Jhn L. Hennessy, Stanfrd University and David A. Pattersn, University f Califrnia at Berkeley

5 On rbustness f datacenters On April 21 st 2011 the Amazn EC2 clud cmputing service was dwn fr 36 hurs taking with it mre than 70 websites, including parts f the New Yrk Times and ppular scial netwrking sites Amazn blamed the prblem n netwrk errrs, but n future details have been disclsed Then, n April 22 nd, Sny Crpratin s PlayStatin Netwrk (PSN), ne f the wrlds largest scial netwrks related t gaming, music, and mvies, was taken ut f service and Sny annunced that the PSN had been cmprmised The PSN was ut f service fr three weeks and arund 77 millin registered users where affected by the utage. Mrever, accunt infrmatin and credit card details fr these users might have been expsed The abve incidents, n tw cnsecutive days, shw real wrld examples f the vulnerability and lack f rbustness in the cmmunicatin technlgy and infrastructure (the clud)

6 Mtivatin Clud cmputing has the ptential t change hw we d cmputing in the future, but fr clud technlgies t reach their full ptential mre research is required. We have identified the fllwing challenges: System utilizatin Rapid elasticity Rbust cmputing Secure cmputing Energy efficiency In additin, this expertise can cmplement the existing plans fr several Nrwegian data centres driven by the Nrwegian industry

7 Execute Research visin Management (SLAs, utilisatin, energy) Cllect perfrmance and faults data. Optimize Mnitr Virtual resurces (VMs, VMMs, IOV etc.) Optimize resurce prvisining based n cllected data and management directives. Create new system cnfiguratin including any manual verrides. Physical resurces (CPUs, RAM, HD, SSD, netwrk devices etc.) Recnfigure resurces.

8 Research tpics Dynamic fabric ruting and resurce allcatin (a self-adaptive netwrk) Methds fr real-time mnitring f cmpute resurces. Autmatically adapt ruting tables, path selectin algrithms, and pririty settings based n existing wrklad and faults. Dynamic prvisining f virtual resurces (a fully virtualized clud) Methds fr real-time mnitring f cmmunicatin resurces. Virtual machine placement and migratin. Imprved virtualizatin f netwrk resurces (e.g. RDMA devices). Increased VM flexibility in rder t rapidly rearrange and scale resurces. Secure strage and sharing (a secure clud, perfrmed by UiS) Guarantee the authenticity f data. Avid unauthrized altering f data. Prevent data leakage when stred at a 3rd party.

9 Clud test bed I We will wrk experimentally and creating prff-cncept systems using: an InfiniBand based private clud a prttype fr each research tpic: a self-adaptive netwrk a fully virtualized clud a secure clud Might add DCB Ethernet t increase technlgical impact

10 Execute Clud test bed II Management (SLAs, OpenStack utilisatin, energy) Cllect perfrmance and faults data. Optimize OFED, OpenSM and new tls Mnitr Virtual resurces Oracle VM/Xen (VMs, VMMs, IOV etc.) Optimize resurce prvisining based n cllected data and management directives. Create new system cnfiguratin including any manual verrides. Physical resurces InfiniBand (CPUs, RAM, HD, SSD, netwrk devices etc.) Recnfigure resurces.

11 Prject details Academic partners: Simula, UiO, and UiS. Industry partners: Oracle and Lyse. Budget f NOK 15.7 millin ver 4 years: 11.5 millin frm the Research Cuncil. 2.6 millins frm Oracle. 1.6 millins frm Lyse. In additin Oracle will cntribute with in-kind There will be 3 Ph.D. students and a pst dc. plus senir persnnel wrking n the prject

12 Nrway well-suited fr hsting green datacenters Cld climate, ffering free cling Empty muntain halls and abandned mines, being bmb prf Cld (sea) water fr effective cling Cmpetitive electricity prices, made frm hydrelectric pwer Stable gelgy Ecnmical and plitical stability Surce: The ICT-Nrway reprt Sky g fjrdane Lefdal mine is green datacenter pilt

13 Thanks fr listening!