Aake Edlund, PhD. KTH and SNIC, Sweden

Transcription

1 Bal4c Cloud Aake Edlund, PhD KTH and SNIC, Sweden Northern Europe Cloud (Project Manager) NEW Bal4cGrid (Project Director) Bal4cGrid Innova4on Lab and Bal4cCloud (Ini<ator, Manager) 1

2 Bal4cGrid the Founda4on Started 1 st of May, 2008, dura<on 2 years Con<nues the Bal<cGrid project ( ) 13 partners from Bal<c States, Belarus, Poland, Sweden, Switzerland Financing 3 MEUR Collabora<ng project for EGEE Suppor<ng NGIs and their collabora<on Normalized compu<ng hours for the Bal<cGrid project and that of the first six months of Bal<cGrid II 2

3 Mo4va4on for Bal4cCloud Startup and prototyping focus When we started, cloud was considered immature, and only for startups and possibly prototyping. This suited us well, knowing that we had not been very successful in a\rac<ng bigger companies. 3

4 The Vision Ini<ated in January 20, 2009 A new way to agract startups A program for early stage companies in our region, by invi<ng them to a hands on course in grid and cloud compu<ng. A course prepared and run by us, on our resources as well as on donated dito from e.g. Amazon. The aim with the course is to help early stage high tech Internet based companies to try their services on new pla_orms, resul<ng in early proof of concepts and later exploita<on of grid and cloud in the region. On top of the course we'll build a network of innova<ve companies in the region. bgin.wordpress.com 4

5 Mo4va4on for Bal4cCloud Low number of applica<ons and applica<on areas in BG. Need of elas<cy volume on demand a way to handle capacity on the fly, to balance the usage Too steep learning curve for grid users Complement to current grid services Low acceptance within industry so far Complicated to adapt new applica<ons, typically, need a project to do that 5

6 Mo4va4on for Bal4cCloud So, one more access point One more, hopefully, easier access point to our Bal<cGrid resources. 6

7 Bal4c Cloud How it all fits together Bal<cGrid BC Planned ac<vi<es SA1 3, NA1 4, JRA BGi Bal<cGrid Innova<on Lab Bal<cCloud Courses SME connec<vity 7

8 Prac4cali4es & Learning by doing Bal<cGrid asked one center per country to install a cloud instance SNIC (Swedish Na<onal Infrastructure for Compu<ng) supported the coordina<on of the work Learning by doing what's in it for escience, really? Is industry interested? Started informa<on gathering and sharing through our site; went to cloud expos and conferences Took early contact with open source cloud vendors, decided to focus on Eucalyptus 8

9 First users Virtual worlds and movies First usage: 1. Virtual world installa<ons (Wonderland), for the Immersive Educa<on organiza<on 2. Film rendering, for Estonian film company July 17,

10 Open source 10 cloud in a box Our (main) choice: - Lead by Rich Wolski, UCSB. - Integrated with - Integrated with BC

11 Major Concerns Will the open source alterna<ves be supported in a sa<sfactory way? Not invented here easily becomes Not supported here Early success open source alterna<ves branching off a private version think Xen, Eucalyptus, which branch will get the best support? Missing standards for interoperability, c f grids. Lock in and de facto lock in, the bidding on a dollar syndrome 11

12 Status Bal4c Cloud See hgp://cloud.bal4cgrid.eu 12

13 Status Estonia NICPB (1/3) KVM based setup with Eucalyptus interface Evalua<ng: other virtualisa<on technologies, e.g. vmware solu<on crea<on of the virtual private networks using Eucalyptus, OpenVPN and dynamic DNS to be able to provide something like the recently rolled out Amazon Virtual Private Cloud (h\p://aws.amazon.com/vpc/) Usage: primarily NICPB ImmersiveEduca<on project with the developement and demo resources for running the Project Wonderland based virtual worlds 13

14 Status Estonia NICPB (2/3) The KVM infrastructure is also used as a pla_orm for the ongoing SME collabora4on on demand render farm crea4on. The plans are to make it controllable by Eucalyptus, but so far it is not capable of suppor<ng Windows machines out of the box (ongoing). Inves4ga4ng the crea4on of the private virtual networks (using OpenVPN and dynamic DNS solu<ons) to be able to provide something like the recently rolled out Amazon Virtual Private Cloud (h\p://aws.amazon.com/vpc/). 14

15 Status Estonia NICPB (3/3) Storage: testbed installa<on of the hadoopfs in NICPB with SRM interfaces. Trying to migrate the storage infrastructure from dcache to HDFS. Wins: easier administra<on, cloud flavor can run hadoop jobs much more efficiently. Problems: Eucalyptus sw is s<ll too alpha/pre beta. The 1.6 _should_ be be\er, incl. support for mul<ple data centers (availability zones) and more robust opera<ons, but it's not out yet. Accoun<ng of consumed resources is complicated the virtualisa<on tools are not making an accent on that. 15

16 Status Latvia RTU Installa4on: Eucalyptus installa<on, xen, 4 cpu for users. HW updates expected (user request). Usage: Submi\ed simple tests, including students tes<ng cloud Issues: System not quite stable. imho so far not usable for serious project. 16

17 Status Belarus UIIP NASB (1/3) Installa4on: HW: 6 CPU, 24 cores (enough for present tasks); Virtualisa<on: Xen; Cloud technolgies: Eucalyptus, Walrus; How to reach: h\ps:// :8443/ Status: 3 GNU/Linux images available, commonly running from 2 to 10 instances. Usage: verified by regular manual monitoring of hosts and services running within the cloud; used by UIIP NASB grid team for designing dynamic grid resource reserva<on and adjustment mechanisms for UNICORE sites 17

18 Status Belarus UIIP NASB (2/3) Usage: pilot project of many users we have prepared a pilot project proposal on deploying one of the advanced informa<on systems of the State Commi\ee for Science and Technologies of the Republic of Belarus (SCST) into the cloud infrastructure provided by the datacenter of the UIIP NASB. The informa<on system to be cloudified is created and administered on behalf of the SCST by Republican Centre for Technology Transfer (h\p:// ic\.by). The system has thousands of visitors and hundreds of ac<ve users who regularly post and update their Technology Offers and Requests, disseminate news, upload technology transfer related applica<ons, graphical and video data. The primary goal of the pilot project is proving the concept of cloud compu<ng being a good way to go not only for SMEs but for some governmental agencies as well. 18

19 Status Belarus UIIP NASB (3/3) Plans: UIIP NASB have plans for cloud usage growth, expansion, connec<on to the outside world and to other countries. [e.g. right now we are buying hardware for the expansion of our cloud infrastructure ] Issues: To increase cloud infrastructure stability we use custom kernel builds for Xen virtual machine images deployed in the cloud. 19

20 Status Lithuania VGTU (1/7) Eucalyptus Cloud Scheme 20

21 Status Lithuania VGTU (2/7) Cloud cluster setup Cluster installa<on is done using Rocks Clusters 5.2 (Chimichanga) VGTU cloud instance is available at h\ps://euca.cloud.vgtu.lt:8443 Eucalyptus v1.5.2 is installed XEN hypervisor is used for paravirtualiza<on MANAGED network mode is configured i.e. machine instances can be accessed directly from internet Available two virtual machine images compiled manually: CentOS 5.3 (Base setup) SLC 53 (Scien<fic Linux Cern) (Base setup) 21

22 Status Lithuania VGTU (3/7) Available Resources Frontend (Cluster controler, Cloud controler): 1 x Pen<um 4 CPU, 2 GB RAM, 200 GB HDD, 1 Gbps Nodes (Node controler) 4 x Pen<um 4 CPU, 2 GB RAM, 200 GB HDD, 1 Gbps 2 x Quad Qore CPU, 4 GB RAM, 320 GB HDD, 1 Gbps Availability zone DEBESELIS Total 12 Cores AVAILABILITYZONE DEBESELIS UP euca.cloud.vgtu.lt AVAILABILITYZONE vm types free / max cpu ram disk AVAILABILITYZONE m1.small 0016 / AVAILABILITYZONE c1.medium 0016 / AVAILABILITYZONE m1.large 0016 / AVAILABILITYZONE m1.xlarge 0016 / AVAILABILITYZONE c1.xlarge 0008 /

23 Status Lithuania VGTU (4/7) Technical ISSUES (problems and limita<ons) In recent eucalyptus version famous error: Error 500 message expired that prevented graphical tools like Elas<cfox, Rightscale, Ylas<c, EC2Dream from working with eucalyptus instance, was not fixed We have fixed this issue manually: Added this code to: HMACQuerySecurityHandler.java as a temporary fix : tsformat.settimezone(timezone.gettimezone("eest")); // Add this Line and import TimeZone class ts.settime( tsformat.parse( <mestamp ) ); And then Eucalyptus rpms were rebuilt. 23

24 Status Lithuania VGTU (5/7) We have installed and use Elas<cFox for Eucalyptus management. 24

25 Status Lithuania VGTU (6/7) We have tried EC2Dream for Eucalyptus management as well. 25

26 Status Lithuania VGTU (7/7) Nearest future Cloud computing activity is included into LitGrid programme Walrus based storage will be installed. Status Sweden PDC Installa<on (ongoing): 4 quad core machines, Eucalyptus, XEN Usage: for tes<ng and learning, to prepare larger installa<on based on older HPC resources 26

27 Status Lithuania VU Installa4on: Small eucaluptus intalla<on (might be increased if demand arrises), xen based, hardware is only 32 bit. Doing some tests with vmware and MS hyper v Usage: Small tests, nothing in produc<on Plans: Thinking of expanding (have to think more about which technology), maybe some services to some other ins<tu<ons Issues: Current system is not very stable, some accidental crashes and system hangings were no<ced. 27

28 Installa4on: Status Poland PSNC two dual CPU machines (one for cloud controller one for cloud nodes), 2x dual core Intel Xeon, 8GM ram/each 147 GB SAS hdd two 1Gbit Eth network card/each virtualiza<on : kvm or xen ( on demand ) cloud stack : eucalyptus url : h\ps://bgcloud.man.poznan.pl:8443, currently available only from PSNC network Usage: test stage Plans for usage and expansion : as part of Bal<c Cloud for local users (PSNC developers and research team), educa<on ins<tu<on in Poznan expansion according to load and u<liza<on we can provide addi<onal cloudnode controllers 28

29 Educa4on VGTU: Bachelor thesis on cloud compu<ng (2009): Deployment of Service Oriented IT Infrastructure VGTU: Cloud compu<ng is part of the MSc course High performance systems at VGTU BG: general presenta<ons, e.g. at ISSGC09 NICPB: A course in IT college (in Tallinn) about the distributed systems. Cloud compu<ng is part of the course, labs will be run on the BC resources NICPB: Tartu cloud and startup school May,

30 Future Expecta4ons on 2009 Connec<ng the Bal<c Cloud with the Northern Europe Cloud Sweden, Norway, Denmark, Estonia, Latvia, Lithuania, Belarus, Poland + Iceland(?) + Availability zones connec<ng the clouds Connec<ng with other clouds, e.g. in the Netherlands See more cloud compu<ng being used by startups and in house quick prototyping See more cloud compu<ng being used in academia esp. HPC flavours See more open source alterna<ves, and alterna<ve cloud providers also in academia Improved solu<ons addressing the dangers in cloud compu<ng. Low expecta<ons on standard APIs July 17,

31 Cloud compu4ng, Virtualisa4on & the Future Summary Bal<cCloud have learned quite much about the current open source cloud technologies what there is to use, what is missing, the quality level The user side is evolving, now that we have some basic courses and an clouds to play with Next level, beyond this pilot project, is to be part of the Northern Europe Cloud 31

32 ? 32

33 Thank you! Aake Edlund (Bal<c Grid) cloud.bal<cgrid.eu (Bal<c Cloud) bgin.wordpress.com (BG Innova<on Lab) 33

34 Basics Cloud Interface Cloud Controller Cluster Controller Node Controller 34

35 Cloud Interface Cloud controller Cloud Controller Cluster Controller Node Controller Web service Java Handles distribu<on of VM instance control events to clusters of resources SLA Engine Cloud controller Instance Scheduler Other Schedulers? 35

36 Cloud Interface Cluster controller Cloud Controller Cluster Controller Web service C based Node Controller Handles distribu<on of VM instance control events to individual of resources Inves<ga<on: what are the performance, invasiveness and func<onality tradeoffs of various cloud networking strategies? Virtual Network Cluster controller Instance Scheduler Other Network/Scheduler solu<ons? 36

37 Cloud Interface Cloud interface Cloud Controller Cluster Controller Web service based Wri\en in Java Handles user event to cloud VM instance control transla<on Experiment: instance crea<on <me Node Controller EC2 compa<ble user interface Cloud interface Web based admin interface Other interfaces? 37