2A SI 6 Ouverture : Stéphane Vialle Stephane.Vialle@supelec.fr http://www.metz.supelec.fr/~vialle Ex. de : 1. Premise of Grid computing 2. Grid computing emergence 3. (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks Premise of Grid Computing CASA project (1990 1995) Objective: distribution of large computations on several supercomputers across a (high speed) Gigabit WAN Two research issues: Impact of high speed WAN on HPC applications? What algorithms and implementations to achieve speedup across a high speed WAN? 1
Chemical computation: Premise of Grid Computing CASA project (1990 1995) Vector computation Calcul 1 Calcul 2 Network Parallel scalar computation Cray YMP Intel DELTA Hiding network time: 3 stage pipeline Overlaped computations and communications t Results: Cray Y MP + Intel DELTA vs Cray Y MP: speedup 3.4 Pipeline: overlapping at 98% Premise of Grid Computing Synthetic Force project (1990 1998) Objective: Distributed Interactive Simulation of War. Distributed Multi Agent system. Military training: Fine simulation of military operations. Many simultaneous users Interactive simulation Research issue: Size Up ( 100) n 100000 entities n 10000 vehicles Premise of Grid Computing Synthetic Force project (1990 1998) Multi Agent simulation: Application with many different computational parts Interactive application with many simultaneous users Interconnection of standard and specific hardware March 1998: simulation of 100000 entities Size up achieved 2
Premise of Grid Computing Results of the experiments Distributed computing across a WAN: Possible: achieves speedup Allows to use specific (adapted) resources (not locally available) Interactive and distributed computing on heterogeneous resources: Possible: achieves size up Allows more realistic simulations, with more users. Main difficulties: complex to develop and to deploy Need of a specific (and new) middleware (grid middleware) Ex. de : 1. Premise of Grid computing 2. Grid computing emergence 3. (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks Ian Foster (Globus 1 st Grid middleware) Ian Foster s view An infrastructure to provide computing/storage/communication capabilities to users, when they need, where they need, and in a transparent way. To build «virtual organizations». 3
Proof of concept (1995 2000) GUSTO: Globus Ubiquitous Supercomputing Testbed Organization 125 sites, 23 countries A testbed for the first Grid middleware Different kinds of Grids Grid of (reliable) supercomputers: for size up to access the right supercomputer (with the right architecture) Different kinds of Grids Grid of available (and volatile) desktop PCs everywhere on Internet: Ex: Seti@Home, LHC@Home, Highly dynamic Grids (resources appear and disappear ) A generic middleware will be developed 4
Different kinds of Grids Management of CERN experiment results Grids of DATA: To share huge data. Data storage + Computing rsrc Repository can not be centralized (too long access times) file migration vs replication? distributed vs centralized directory? directory and repository coherence? Different kinds of Grids Collaborative Grid: distributed virtual and enhanced reality Real time issues, requires QoS Concert across Internet in 2003, with expensive devices Collaborative work on virtual objects in 2009, medium cost devices Ex. de : 1. Premise of Grid computing 2. Grid computing emergence 3. (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks 5
Key issues of a Grid middleware (1) Adaptive applications high level services low level modules Distributed resources The «hourglass» model (of Globus 1): Adaptability is required to maintain functionalities when the Grid is changing Global (collective) services Ex : authentication all over the Grid Minimal set of available (and standard) functionalities on each resource Local functionalities on the resources with standard interfaces Ex : authentication on the local resource Heterogeneous rsrc and many potential failures Key issues of a Grid middleware (2) From high level services to low level modules Each site has its own authentication mechanisms can be long and complex to log on each site! A user authenticate only once, using a high level authentication service (with a std interface) Then he is automatically authenticated on any resources he is using, with the help of low level modules Adaptive applications High level services Low level modules Distributed resources Key issues of a Grid middleware (3) 3 fundamental hourglasses + a security base LDAP based (hierarchy of LDAPs) GridFTP (extension of «ftp») + Grid directory Brooker and scheduler + DataBase of resource availability Based on certificat exchange «Without security issues, there would be no Grid middleware research». G. Khan, 2004. 6
Becoming closer of Web services Applications Open Grid Service Architecture GLOBUS III GLOBUS IV using web services GLOBUS IV evolution OGSA Architected Services Applications OGSI Open Grid Services Infrastructure OGSA Architected Services Web Services OGSA Enabled OGSA Enabled Security Workflow OGSA Enabled s OGSA Enabled OGSA Enabled Database File Systems OGSA Enabled Storage OGSA Enabled OGSA Enabled Directory Messaging OGSI Open Grid Services Infrastructure OGSA Enabled Web Services Network Web Services WSRF OGSA Enabled OGSA Enabled Security Workflow OGSA Enabled s OGSA Enabled OGSA Enabled Database File Systems OGSA Enabled Storage OGSA Enabled OGSA Enabled Directory Messaging OGSA Enabled Network Grid middleware for independent tasks PC cluster (workers) Client running: embarrassingly parallel applications, or sequential tasks XtremWeb server Internet Desktop PCs (clients or workers) Key issues: to deal with security of desktop workers support high volatility of desktop workers Many similar industrial products and Grid middleware for independent tasks Communication protocol Worker XtremWeb server: communication started by the worker data sent back to the worker by the server User/Worker Becoming unused Waiting for a task Processing a task Task finished Rq : hostregister Contact the last known XtremWeb server, or contact the root XtremWeb server Rq : workrequest Send its features to the XtremWeb server and wait for a task to process Rq : workalive Signal its activity to its XtremWeb server Rq : workresult Send its result to a specified server or to its XtremWeb server Compatible with stateless firewalls Task server Register the new worker Send back a task: binary code, data, server @ (to reply) Re schedule the task if no I am alive message received 7
Network Enabled (NES): an API and some Grid middlewares for RPC //GridRPC grpc_call User application API: GridRPC User laptop Brooker («agent») allocating resources + Specific Grid middleware + (NetSolve, Ninf, Diet) Computing servers (running «solvers») Distributed resources Allows to implement more complex parallel applications: can call a parallel solver (run on one site) can overlap solver runs and/or communications (async. RPC) Network Enabled (NES): an API and some Grid middlewares for RPC Generic mechanism for resource allocation: On demand analyse, allocation and scheduling 1 2 Client Scheduler 4 3 Database Monitor Monitor Monitor Components of the brooker (or «agent») Continuous performance measurement Statistical DB building 5 API and Grid middleware for Active Objects JAVA based Support a large variety of parallel algorithms: Independent computations GRID RMI (object version of GRID RPC) Master Workers Parallel algorithms with data circulation 8
API and Grid middleware for a virtual shared memory JavaSpace A virtual shared memory space is located on one machine. A space is R/W by any task from any machine. Industrial products with distributed spaces (scalable spaces) exist : Ex. de : 1. Premise of Grid computing 2. Grid computing emergence 3. (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks Evolution of Grids from 1998 to 2006 HPC meta computing Web services e Science Grids Experimental Grids Enterprise Grids 3 families of Grids 9
e Science Grids 2008: 259 sites 52 countries EGEE: European production Grid for scientific research» Middleware: Globus II Globus II ++ glite 2008: 150/200 Virtual Organizations 7500/14000 users 150000 jobs/day Each participant brings resources, and can access all resources. Grid engineers make compatible security policies of all sites Experimental Grids Objective 1: 5000 processors objective: 5000 cores achieved, and surpassed Objective 2: a real multi site Grid 15 clusters on 9 sites, +++ Private RENATER network, 10Gbit/s Objective 3: a Grid for computer science research NO production of applicative results Grid 5000: French experimental Grid On demand OS deployment!! «first cloud in the world» Enterprise Grids Definition: One enterprise uses its PCs to run its independent & compute intensive tasks. The enterprise Grid is installed inside its secure network. PCs of the Grid are not volatile: desktop PCs: PC of absent people, server farm: reliable and monitored. Examples: September 2006: BNB Paribas used a 3000 server PC Grid June 2007: PSA observed great success of its 300 desktop PC Grid (with QoS) (low QoS) 10
Ex. de : 1. Premise of Grid computing 2. Grid computing emergence 3. (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks Voluntary computing + unused PCs in the laboratories Example of structured Grid New e Science Grid for LHC CERN LHC A World Grid both for data and computations: 15 PetaOctets/year Huge computations LHC@home LCG: LHC Computing Grid Example of structured Grid New e Science Grid for LHC LCG: the LHC Computing Grid A hierarchy of tier nodes, to store and cache huge data, and to run huge computations. Node T 0 : data production site, at CERN Node T 1 : Complete storage sites in the world Node T2 : Partial storage site in the world (database caches) Node T3 : Computing nodes in the world (ex: clusters) 11
Ex. de : 1. Premise of Grid computing 2. Grid computing emergence 3. (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks Cloud Computing Emergence An industrial initiative (?) Business model: A company provides computing resources Price is function of the consumed CPU time (parallelism is free) Data security inside the cloud Secure connection to join the cloud QoS is provided Virtualization on each node allows to support various applications from various customers Personal point of view: the next step after Enterprise Grids. Cloud Computing Emergence An industrial initiative (?) Large data centers: have global cooling and control of the machines allows to reduce management cost Mutualisation: a customer requires machines only when he needs (on demand) allows to share resources between users (customers) Ex : in March 2009, Salesforce.com managed data of 54 000 entreprises with only 1000 servers. Virtualization: each application is run inside a virtual machine provides the environment required by the customer application allows to always use the most efficient available machines allows to use at 100% each machine (if machine are shared) Data centers, Virtualization and Mutualisation reduce the cost 12
Cloud Computing Emergence A basic classification Cloud of software (Software as a Service) Ready to run some identified applications Ex: Supelec experimented a private cloud of Matlab Cloud of plateform (Platform as a Service) Ready to provide a development environment on an available hardware resource. Google search engine? Google mail? Grid 5000? Cloud of hardware (Infrastructure as a Service) Ready to provide hardware resources. The user/customer has to deploy an environment before to use these machines. Amazone? Cloud Computing Emergence First cloud assessment «CLOUD» buzzword is a success! Several major actors make money with «cloud technology» Cloud was initially less ambitious than Grid, but aimed to be easier to deploy, to manage and to use From a customer point of view : using extra resources on demand is a nice idea how can I be sure to get these extra resources when I need? limit of the «mutualisation» and QoS how can I be sure my data are in a secure system? limit of virtualization, remote security and transfer security Private cloud can be an interesting compromise Cloud Computing Emergence From Grid to Cloud? HPC Web services e Science Grids Experimental Grids Enterprise Grids SaaS PaaS Virtualisation technics IaaS Security 3 kinds of Clouds 13
Ex. de : 1. Premise of Grid computing 2. Grid computing emergence 3. (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks New Cloud Computing Approach New approach in cloud computing New cloud researches: Multi site Clouds Inter Clouds (interconnecting clouds of different institutions) fusion of different cloud providers a cloud provider can rent resources of another one But looks like a Grid? Grid middleware technology is now introduced has the «Grid infrastructure for Cloud» (HPCS 2011 conference) 50% news about «cloud» claim this new approach is promising 50% news about «cloud» say «WARNING»! it will (technically) fail! there is no business model! there is no need for these new clouds! New Cloud Computing Approach New classification The NIST definition of cloud computing (September 2011): Axis 1: IaaS, PaaS, SaaS Axis 2: who can use the cloud? Private cloud: exclusive use by a single organization comprising multiple consumers (business units) Community cloud: exclusive use by a specific community of consumers from different organizations Public cloud: open use by the general public Hybrid cloud: composition of 2 or more cloud infrastructures (private, public, or community) Rmk: does not focus on cloud owner or manager! Just on users. 14
New Cloud Computing Approach New classification Possible complete classification: Users Where are stored the data? Ex: EU data have to stored inside EU Hybrid clouds Public clouds Community clouds Private clouds Owner & manager (?) Any institutions Some limited institutions One limited institution Institution of the only user Service IaaS PaaS SaaS New Cloud Computing Approach Future of the Cloud Computing 2008 2010: Europe has started some FP7 projects about «cloud» development of new Cloud middleware some of these projects are now calling for experiments currently no real evaluation September 2011: Europe opened a new call about «cloud» to not develop new middleware to develop «cloud applications» : with a business model at European scale 2012 : Governments want to be sure institutional/national data are stored inside the country! Pb with inter clouds/cloud roaming New Cloud Computing Approach Future of the Cloud Computing Questions : Missing real large cloud applications? A European cloud infrastructure has a sense? Multi sites, multi institution clouds have a sense? HPC Cloud? Aneo & Supelec are experimenting the new MS Azur HPC cloud 15
Ex. de : 1. Premise of Grid computing 2. Grid computing emergence 3. (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks Réseaux de capteurs Des avec de nouvelles contraintes Motivation : un moyen de «réaliser des mesures au plus près d un phénomène» Aujourd hui : de k 100 à k 1000 capteurs. demain : k 100 000. Différentes configurations avec différentes contraintes Alimentation électrique pérenne Batteries ou alimentation non pérenne Capteurs fixes Ex : tracking en environnement urbain ou industriel Ex : mesure de température dans la nature Capteurs mobiles Ex : capteurs embarqués dans des voitures Ex : capteurs portés par des personnes Réseaux de capteurs Des applications très variées Exemples d applications : Mesure du graphe de mobilité des personnes dans un hôpital pour détecter des contaminations et propagation de maladies nosocomiales. Collecte de mesure environnementales (températures, pluie) et comportementale (allumage phares, déclenchement d ABS) par des capteurs dans les voitures, pour améliorer la gestion du trafic. Supervision de la radioprotection en temps réel dans les centrales nucléaires (EDF) par des réseaux ad hoc de capteurs sans fil. 16
Réseaux de capteurs Besoin spécifiques aux réseaux de capteurs Besoin de nouveaux protocoles réseaux pour : minimiser les dépenses d énergie tolérer de nombreuses pannes de nœuds du réseau supporter l apparition et la disparition fréquente de nœuds mobiles Besoin de traitement de signal complexe pour : analyser de gros flux d information détecter des données erronées Besoin de nouvelles applications distribuées pour : exploiter ces masses de données en temps réel réagir rapidement en cas de situation anormale Exemple : projet Live E! En février 2009 : 106 stations météo dans13 pays From Grid to private Clouds, to inter Clouds. End 17