Modalis. Des grilles aux clouds : Hélène Renard, I3S - UMR 6070 UNSA / CNRS - nouveaux problèmes, nouvelles solutions

Transcription

1 Journée Des grilles aux clouds : nouveaux problèmes, nouvelles solutions Hélène Renard, I3S - UMR 6070 UNSA / CNRS - École polytechnique universitaire de Nice-Sophia Antipolis 23 février Journée Des grilles aux clouds Modalis

2 Hélène Renard Des grilles aux clouds 2 Plan de la présentation 1. Évolution ou révolution? (Thierry Priol) Virtuous side Dark side Conclusion 2. Sky Computing on FutureGrid and Grid 5000 with Nimbus (Pierre Riteau) Introduction to Sky Computing The Nimbus Project Large-Scale Sky Computing Experiments Conclusion 3. Des grilles vertes aux nuages verts : vers des systèmes efficaces en énergie (Laurent Lefèvre) 4. A Telecommunications Operator View on Cloud Computing (Thierry Coupaye) 5. Questions éventuelles

3 Évolution ou révolution? Plan de la présentation Hélène Renard Des grilles aux clouds 3 1. Évolution ou révolution? (Thierry Priol) Virtuous side Dark side Conclusion 2. Sky Computing on FutureGrid and Grid 5000 with Nimbus (Pierre Riteau) Introduction to Sky Computing The Nimbus Project Large-Scale Sky Computing Experiments Conclusion 3. Des grilles vertes aux nuages verts : vers des systèmes efficaces en énergie (Laurent Lefèvre) 4. A Telecommunications Operator View on Cloud Computing (Thierry Coupaye) 5. Questions éventuelles

4 Hélène Renard Des grilles aux clouds 4 Évolution ou révolution? Grilles vs Clouds Virtuous side Définition de la grille A fully distributed, dynamically reconfigurable, scalable and autonomous infrastructure to provide location independent, pervasive, reliable, secure and efficient access to a coordinated set of services encapsulating and virtualizing resources (computing power, storage, instruments, data, etc.) in order to generate knowledge... from the CoreGRID NoE

5 Hélène Renard Des grilles aux clouds 5 Évolution ou révolution? Grilles vs Clouds Virtuous side Définition du cloud An emerging computing paradigm where applications, data and infrastructures are provided as a service that can be ubiquitously accessed from any connected devices over the internet.

6 Évolution ou révolution? What is behind Cloud Virtuous side Google cluster 1997 Datacenters as the reincarnation of the mainframe concept The end of the PC/Mac era? Just a web browser is needed The network is the computer, thin client,... Hélène Renard Des grilles aux clouds 6

7 Hélène Renard Des grilles aux clouds 7 Évolution ou révolution? Virtuous side Why Cloud now and not before? Internet! Network performance has been improved dramatically the last 15 years Nearly always connected to the Internet (anytime, anywhere) PC is not anymore the central device for personal computing MP3, SmartPhone, Tablets, Set-top box, PCs,... How to get access to my personal data anywhere/anytime and from any devices? Cost Oversized systems to meet peak demand (both in the private and public sector) Outsourcing (labor cost is much higher that computing cost)

8 Hélène Renard Des grilles aux clouds 8 Évolution ou révolution? Virtuous side Computing as a utility : a brief history

9 Hélène Renard Des grilles aux clouds 9 Évolution ou révolution? Cloud Acronyms Virtuous side PaaS - Platform/People as a Service SaaS - Software/Search as a Service IaaS - Infrastructure as a Service DaaS - Data as a Service CaaS - (composition/communication /composite) as a Service HaaS - Human as a Service... just your shared agenda ;-)... KaaS - Knowledge as a Service

10 Hélène Renard Des grilles aux clouds 10 Évolution ou révolution? Virtuous side Cloud : how to escape from the jungle

11 Hélène Renard Des grilles aux clouds 11 Évolution ou révolution? Virtuous side Infrastructure as a Service Get access on demand to a large number of highly virtualized resources Dynamicity, elasticity. Concept of OS Virtualization OS does not matter anymore! OS are just software libraries and does not play a central role! Concept of virtual machines to host instances of OS ; Physical resources are shared by several virtual machines.

12 Hélène Renard Des grilles aux clouds 12 Évolution ou révolution? Virtuous side Let s take an example... Amazon!

13 Hélène Renard Des grilles aux clouds 13 Évolution ou révolution? Platform as a Service Virtuous side An application development, deployment and management fabric. User programs web service front end and computational & Data Services Framework manages deployment and scale out No need to manage VM images

14 Hélène Renard Des grilles aux clouds 14 Évolution ou révolution? Virtuous side Software as a Service

15 Hélène Renard Des grilles aux clouds 15 Évolution ou révolution? Dark side What is the dark side of Cloud computing? We have only seen the virtuous side! What is the dark side of Cloud Computing?

16 Hélène Renard Des grilles aux clouds 16 Évolution ou révolution? Dark side Some research issues with Cloud Computing Reliability / Resilience / Fault-tolerance ; Trust, Security and Privacy ; New economical models for computing ; Service Level Agreement / Quality of Service ; Building cloud-aware applications from legacy applications ; Energy management ; Data management ; Cloud federation ; Autonomic behaviors / Self-* ; Brokering / Scheduling ; Programming models (MapReduce,...) ; Interactions between legal aspects (laws) and computer science : privacy and liability

17 Hélène Renard Des grilles aux clouds 17 Évolution ou révolution? Dark side Are Cloud infrastructures less secure than non-cloud ones? One of the fastest and easiest ways to access corporate data is through unprotected PDAs that are lost or stolen, as they contain business names and addresses, spreadsheets and other corporate documents pda sec 60% of corporate data resides unprotected on PC desktops and laptops (IDC analyst Cynthia Doyle, Business Continuity in 2002 : It s Not Business as Usual, April 2002) Read from 10% of laptop computers will be stolen within the first 12 months of purchase. 90% of stolen laptops are never recovered. 49% of companies have had laptops stolen with the last 12 months. 57% of corporate crimes are linked to stolen laptops. 80% of computer crime consists of inside jobs by disgruntled employees. 73% of companies had no specific security policies for their laptops in % of USB thumb drive owners report losing them, over 60% with private corporate data on them!

18 Hélène Renard Des grilles aux clouds 18 Conclusion Évolution ou révolution? Conclusion Cloud is becoming a buzzword... a lot of hype around it : Not the swiss knife for distributed computing ; More an evolution than a revolution ; Less ambitious than Grid but there is an increasing public and business demand. But there are new opportunities for research : Reliability / Resilience / Fault-tolerance ; Trust, Security and Privacy ; New economical models for computing ; Service Level Agreement / Quality of Service ; Building cloud-aware applications from legacy applications ; Energy management ; Cloud federation ; Autonomic behaviors / Self-* ; Brokering / Scheduling (performance, energy,...) ; Programming models (MapReduce,...) ; Interactions between legal aspects (laws) and computer science.

19 Sky Computing on FutureGrid and Grid 5000 with Nimbus Plan de la présentation Hélène Renard Des grilles aux clouds Évolution ou révolution? (Thierry Priol) Virtuous side Dark side Conclusion 2. Sky Computing on FutureGrid and Grid 5000 with Nimbus (Pierre Riteau) Introduction to Sky Computing The Nimbus Project Large-Scale Sky Computing Experiments Conclusion 3. Des grilles vertes aux nuages verts : vers des systèmes efficaces en énergie (Laurent Lefèvre) 4. A Telecommunications Operator View on Cloud Computing (Thierry Coupaye) 5. Questions éventuelles

20 Sky Computing on FutureGrid and Grid 5000 with Nimbus Outline Hélène Renard Des grilles aux clouds 20 Introduction to Sky Computing The Nimbus Project Large-Scale Sky Computing Experiments Conclusion

21 Hélène Renard Des grilles aux clouds 21 Sky Computing on FutureGrid and Grid 5000 with Nimbus Introduction to Sky Computing Infrastructure-as-a-Service Basic features Run VM from VM image Modify + save VM image Terminate VM All operations accessible through an API Autonomic infrastructure management Business model Pay for CPU time + network traffic in/out + storage Initially targeting web service hosting Also HPC now (Amazon Cluster Compute instances)

22 Hélène Renard Des grilles aux clouds 22 Sky Computing on FutureGrid and Grid 5000 with Nimbus Sky Computing Introduction to Sky Computing Federation of multiple clouds Creates large scale infrastructures Allows to run software requiring large computational power

23 Hélène Renard Des grilles aux clouds 23 Sky Computing on FutureGrid and Grid 5000 with Nimbus The Nimbus Project The Nimbus Project Started in 2005 by Kate Keahey (Argonne/UC) Cloud computing Toolkit Open source IaaS implementation Amazon EC2-compatible (WSDL and Query APIs) Cumulus (Amazon S3-compatible storage cloud) Targets Clouds for Science Unique features Context Broker Workspace Pilot LANTorrent Spot instances

24 Hélène Renard Des grilles aux clouds 24 Sky Computing on FutureGrid and Grid 5000 with Nimbus The Nimbus Project The Nimbus Architecture

25 Hélène Renard Des grilles aux clouds 25 Sky Computing on FutureGrid and Grid 5000 with Nimbus Context Broker The Nimbus Project Service to configure a complete cluster with different roles Works with a cluster distributed on multiple clouds (e.g. Nimbus and Amazon EC2) VMs contact the context broker to Learn their role Learn about other VMs in the cluster Ex. : Hadoop master + Hadoop slaves Hadoop slaves configured to contact the master Hadoop master configured to know the slaves

26 Hélène Renard Des grilles aux clouds 26 Sky Computing on FutureGrid and Grid 5000 with Nimbus Large-Scale Sky Computing Experiments Large-Scale Sky Computing Experiments Nimbus Resource management Contextualization ViNe All-to-all connectivity Hadoop Task distribution Fault tolerance Resource dynamicity

27 Hélène Renard Des grilles aux clouds 27 Sky Computing on FutureGrid and Grid 5000 with Nimbus ViNe Large-Scale Sky Computing Experiments Project of the University of Florida (M. Tsugawa et al.) High performance virtual network All-to-all connectivity

28 Hélène Renard Des grilles aux clouds 28 Sky Computing on FutureGrid and Grid 5000 with Nimbus Hadoop Large-Scale Sky Computing Experiments Open-source MapReduce implementation Heavy industrial use (Yahoo, Facebook...) Efficient framework for distribution of tasks Built-in fault-tolerance Distributed file system (HDFS)

29 Hélène Renard Des grilles aux clouds 29 Sky Computing on FutureGrid and Grid 5000 with Nimbus Grid 5000 Overview Large-Scale Sky Computing Experiments Distributed over 9 sites in France 1500 nodes, 5500 CPUs Study of large scale parallel/distributed systems Features Highly reconfigurable Environment deployment over bare hardware Can deploy many different Linux distributions Even other OS such as FreeBSD Controlable Monitorable (metrics access) Experiments on all layers Network, OS, middleware, applications

30 Hélène Renard Des grilles aux clouds 30 Sky Computing on FutureGrid and Grid 5000 with Nimbus Large-Scale Sky Computing Experiments FutureGrid : a Grid Testbed NSF-funded experimental testbed 5000 cores 6 sites connected by a private network

31 Hélène Renard Des grilles aux clouds 31 Sky Computing on FutureGrid and Grid 5000 with Nimbus Large-Scale Sky Computing Experiments Resources used in Sky Computing Experiments 3 FutureGrid sites (US) with Nimbus installations UCSD (San Diego) UF (Florida) UC (Chicago) Grid 5000 sites (France) Lille (contains a white-listed gateway to FutureGrid) Rennes, Sophia, Nancy, etc. Grid 5000 is fully isolated from the Internet One machine white-listed to access FutureGrid ViNe queue VR (Virtual Router) for other sites

32 Hélène Renard Des grilles aux clouds 32 Sky Computing on FutureGrid and Grid 5000 with Nimbus Experiment scenario Large-Scale Sky Computing Experiments Hadoop shy virtual cluster already running in FutureGrid (SD, UF, UC) Launch BLAST MapReduce job Start VMs on Grid 5000 resources With contextualization to join the existing cluster Automatically extend the Hadoop cluster Number of nodes increases TaskTracker nodes (Map/Reduce tasks execution) DataNode nodes (HDFS storage) Hadoop starts distributing tasks in Grid 5000 Job completes faster!

33 Hélène Renard Des grilles aux clouds 33 Sky Computing on FutureGrid and Grid 5000 with Nimbus Conclusion Conclusion Sky Computing to create large scale distributed infrastructures Our approach relies on Nimbus for resource management, contextualization and fast cluster instantiation ViNe for all-to-all connectivity Hadoop for dynamic cluster extension Provides both infrastructure and application elasticity

34 Des grilles vertes aux nuages verts Plan de la présentation Hélène Renard Des grilles aux clouds Évolution ou révolution? (Thierry Priol) Virtuous side Dark side Conclusion 2. Sky Computing on FutureGrid and Grid 5000 with Nimbus (Pierre Riteau) Introduction to Sky Computing The Nimbus Project Large-Scale Sky Computing Experiments Conclusion 3. Des grilles vertes aux nuages verts : vers des systèmes efficaces en énergie (Laurent Lefèvre) 4. A Telecommunications Operator View on Cloud Computing (Thierry Coupaye) 5. Questions éventuelles

35 Des grilles vertes aux nuages verts Energy : 1st challenge for large scale systems? Hélène Renard Des grilles aux clouds 35 Future exascale platforms -> systems from 20 to 100MW (current 4-6 MW) How to build such systems and make them energy sustainable / responsible? Hardware can help (component by component) Software must be adapted to be scalable but also more energy efficient Usage must be energy aware

36 Des grilles vertes aux nuages verts Power demand and Green IT explosion Hélène Renard Des grilles aux clouds 36 IT : 2-5% of CO2 emissions Green It -> reducing electrical consumption of IT equipments - CO2 impact depends on countries Focus on usage : fighting un-used/over- provisioned plugged resources Problem : grey energy (ecoinfo.org) GreenIT scientific events (12/ / tracks/issues greenitconferences.org)

37 Des grilles vertes aux nuages verts Green-IT Leverages Hélène Renard Des grilles aux clouds 37 Shutdown : reducing the amount of powered unused resources Slowdown : adapting the speed of resources to real usage Optimizing : improving hardware and software for energy reduction purpose Coordinating : using large scale approaches to enhance green leverages

38 Des grilles vertes aux nuages verts Methodology Hélène Renard Des grilles aux clouds 38 Proposing a generic energy aware model able to be derivated onto different scenario (Grids, Clouds, Networks) Designing software solutions for infrastructures Simulating and Validating at medium and large scale

39 Des grilles vertes aux nuages verts Conclusion Hélène Renard Des grilles aux clouds 39 At the beginning, the Grid was green (cycle stealing, old machines usage...) like the Cloud (aggregation) Energy aspects change the way we design Grids&Clouds applications, protocols, services and policies (i.e. load balancing is not always the best solution) Challenge : design energy proportional equipments and frameworks (computing, memory or network usage)

40 A Telecommunications Operator View Plan de la présentation Hélène Renard Des grilles aux clouds Évolution ou révolution? (Thierry Priol) Virtuous side Dark side Conclusion 2. Sky Computing on FutureGrid and Grid 5000 with Nimbus (Pierre Riteau) Introduction to Sky Computing The Nimbus Project Large-Scale Sky Computing Experiments Conclusion 3. Des grilles vertes aux nuages verts : vers des systèmes efficaces en énergie (Laurent Lefèvre) 4. A Telecommunications Operator View on Cloud Computing (Thierry Coupaye) 5. Questions éventuelles

41 A Telecommunications Operator View Agenda Hélène Renard Des grilles aux clouds 41 An industrial view on cloud computing Drivers and barriers associated to cloud computing Orange & cloud computing Stakes & current offers A glimpse of the future at Orange Labs The open cloud, the personal cloud, cloud networking... The last word, cloud and grid?

42 A Telecommunications Operator View Drivers for users Hélène Renard Des grilles aux clouds 42 Cost savings Reduced TCO : hardware, software, IT staff Pay-per-use (pay as you go), multitenancy Simplicity, Faster Time-to-market Easy service experimentation before service launch Faster deployment, no need for servers and software to launch a service adflexibility, Scalability Automatic, transparent scale up and down Improved availability and QoS Although well-known breakdowns, cloud services have statistically better availability than on site services... and they are improving

43 A Telecommunications Operator View Barriers Hélène Renard Des grilles aux clouds 43 Lack of technical maturity SLA, auto-scaling/auto-sharing, performance, availability, dependability PaaS : applications packaging, deployment, management, test, configuration management... Storage Network Security, privacy withmultitenancy Major risks of lock-in lock-in Lack of standards (API, programming models) Lack interoperability Lack of portability (applications,mgt tools) Legal Issues Software licences Data location (eg government, health) Integration with legacy IT (IS) Huge investments in data centers building, hardware, cooling, energy

44 A Telecommunications Operator View France Télécom / Orange Hélène Renard Des grilles aux clouds 44

45 A Telecommunications Operator View The Future of Orange Labs Hélène Renard Des grilles aux clouds 45

46 A Telecommunications Operator View The last word : scalability Hélène Renard Des grilles aux clouds 46 Sclability in mainstream cloud is currently more perceived : as hundreds of thousand applications/users each having a few virtual machines than about few applications having each thousands of virtual machines What about then : Grid, HPC & cloud Cloud for e-science? Cloud market for e-science?

47 Questions éventuelles Plan de la présentation Hélène Renard Des grilles aux clouds Évolution ou révolution? (Thierry Priol) Virtuous side Dark side Conclusion 2. Sky Computing on FutureGrid and Grid 5000 with Nimbus (Pierre Riteau) Introduction to Sky Computing The Nimbus Project Large-Scale Sky Computing Experiments Conclusion 3. Des grilles vertes aux nuages verts : vers des systèmes efficaces en énergie (Laurent Lefèvre) 4. A Telecommunications Operator View on Cloud Computing (Thierry Coupaye) 5. Questions éventuelles

48 Questions éventuelles Questions éventuelles Hélène Renard Des grilles aux clouds 48 Le pire, quand on fait un discours, ce n est pas de prendre conscience que vos auditeurs regardent leur montre, c est le moment où ils se mettent à la secouer pour voir si elle n est pas arrêtée. C est à vous!