Cloud Computing mit mathematischen Anwendungen

Transcription

1 Cloud Computing mit mathematischen Anwendungen Vorlesung SoSe 2009 Dr. Marcel Kunze Karlsruhe Institute of Technology (KIT) Steinbuch Centre for Computing (SCC) KIT the cooperation of Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)

2 Agenda Cloud Computing 1. Einleitung Was ist Cloud Computing? 2. Grundlagen Virtualisierung, Web Services, 3. Cloud Architekturen Infrastruktur, Plattform, Anwendung 4. Cloud Services Amazon Web Services, Google App Engine 5. Aufbau einer Cloud OpenCirrus Projekt, Eucalyptus, Hadoop 6. Cloud Algorithmen MapReduce, Optimierungsverfahren, Praktische Übungen und Anwendungen Vorlesung im Web: 2 2

3 Commercial Cloud Offerings (Small Excerpt) Problem: Commercial offerings are proprietary and usually not open for cloud systems research and development 3

4 Cloud Systems Research Simple, transparent, controllable cloud computing infrastructure What types of interfaces are appropriate for clouds? How should cloud networks be constructed/managed? How are security concerns addressed in the cloud? How are various workloads most efficiently transferred? What types of applications can run in clouds? What types of service level agreements are appropriate/possible? Research requirements Perform experiments also on a low system level Flexible cloud computing framework Compare different methodologies and implementations 4

5 Cloud Computing OpenCirrus TM Cloud computing R&D: OpenCirrus TM project 5

6 6

7 OpenCirrus Cloud Computing Research Testbed An open, internet-scale global testbed for cloud computing research Data center management & cloud services Systems level research Application level research Structure: a loose federation Sponsors: HP Labs, Intel Research, Yahoo! Partners: UIUC, Singapore IDA, KIT, NSF Members: System and application development Great opportunity for cloud R&D 7

8 Where are the OpenCirrus sites? Six sites initially: Sites distributed world-wide: HP Research, Yahoo!, UIUC, Intel Research Pittsburgh, KIT, Singapore IDA processor cores per site New CMU site coming in 2009 HP Yahoo (sf) UIUC Intel (pgh) CMU (coming in 09) KIT (de) IDA (sg) 8

9 HC3: OpenCirrus Installation at KIT (July) Infniband QDR Interconnect MSA2000fc 2656 Nehalem Cores / 332 Servers / 10 TB Memory Infiniband QDR Interconnect 192 TB Lustre Filesystem 9

10 OpenCirrus Global Services Federation of sites needs common global services Global SignOn Single authentication at all sites Based on distribution of ssh keys Global Monitoring Based on Ganglia ( Scalable distributed monitoring system Supports federations of clusters OpenCirrus service at Global Data TBD 10

11 Cloud Architecture Source: S.Tai 11

12 OpenCirrus TM Blueprint Cloud application services Virtual Resource Sets Cloud infrastructure services Eucalyptus IT infrastructure layer (Physical Resource Sets) 12

13 Physical Resource Sets (PRS) PRS service goals Provide mini-datacenters to researchers Isolate experiments from each other Stable base for other research PRS service approach Allocate sets of physical co-located nodes, isolated inside VLANs. Leverage existing software (e.g. Utah Emulab, HP OpsWare) Start simple, add features as we go Base to implement virtual resource sets Hardware as a Service (HaaS) 13

14 Virtual Resource Sets (VRS) Basic idea: Abstract from physical resource by introduction of a virtualization layer Concept applies to all IT aspects: CPU, storage, networks and applications, Main advantages Implement IT services exactly fitting customer s varying need Deploy IT services on demand Automated resource management Easily guarantee service levels Live migration of services Reduce both: CapEx and OpEx Infrastructure as a Service (IaaS) Implement Compute and Storage services De-facto standard: Amazon Web Services interface 14

15 Eucalyptus: A potential VRS Layer EUCALYPTUS - Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems. Open-Source software infrastructure for implementing cloud computing on clusters from UC Santa Barbara. Impelements Infrastructure as a Service (IaaS) gives the user the ability to run and control entire virtual machine instances (Xen) deployed across a variety of physical resources. Interface compatible with Amazon EC2 and S3. Potential to interact with the same tools, known to work with Amazon EC2 and S3. Linux image hosting ala Amazon Web Services. Eucalyptus is an important step for archiving an open and established cloud computing infrastructure standard. Spin-off Eucalyptus Systems with venture capital 15

16 Eucalyptus: A potential VRS Layer Schedules the distribution of virtual machines to the NC. Collects (free) resource information. Collects resource information from the CC. Operates like a meta-scheduler in the cloud. Runs on every node in the cloud. Xen- Hypervisor running. Provides Information about free resources to the CC. Source: R.Wolski 16

17 R&D Cloud Installation at SCC/KIT Eucalyptus R&D Cloud I 2x IBM Blade LS20 Dual Core Opteron (2,4GHz) 4GB RAM 2x IBM Blade HS21 Dual Core Xeon (2,33GHz) 16GB RAM Eucalyptus R&D Cloud II 5x HP Blade ProLiant BL2x220c Each Blade: 2 Server Nodes 2x Intel Quad-Core Xeon (2,33GHz) 16GB RAM 17

18 Comparing Storage Performance between S3 and Eucalyptus WOW! Sequential Output Per-Character: file is written usind putc() Block: file is written using write() Rewrite: read() and write() Sequential Input Per-Character: file is read usind getc() Blockwise: file is written using read() Source: C.Baun 18

19 Realistic values The RAM of the Eucalyptus Node Controller was reduced to overcome memory caching. The storage performance of Eucalyptus depends on the available storage sub-system. Source: C.Baun 19

20 Performance of Random Seeks and File Creation The RAM of the Eucalyptus Node Controller was reduced to overcome memory caching. The storage performance of Eucalyptus depends on the available storage sub-system. Source: C.Baun 20

21 How is OpenCirrus different from other testbeds? OpenCirrus TM supports both system- and app-level research n/a at Google/IBM and EC2/S3 OpenCirrus TM researchers will have complete access to the underlying hardware and software platform. OpenCirrus TM allows Intel platform features that support cloud computing (e.g. DCMI, NM) to be exposed, and exploited. Map-Reduce apps Hadoop Virtual machines Google/IBM cluster Map-Reduce apps Hadoop Cluster mgmt software Virtual or physical machines Open Cirrus cluster Can be modified by users Cannot be modified by users Cloud apps and services Can be modified by users 21

22 Summary OpenCirrus TM offers interesting R&D opportunities Cloud systems development Cloud application development Accepting research proposals OpenCirrus TM workshop at HP Palo Alto on June 8/9 22

23 Karlsruhe Institute of Technology Steinbuch Centre for Computing (SCC) Thank you for your attention. 23