Cloud-Based Computation and Collaboration: the Challenges for IT Infrastructure. Greg Bell Driving e-research Across the Pacific `09 Sydney: 11/12/09

Transcription

1 Cloud-Based Computation and Collaboration: the Challenges for IT Infrastructure Greg Bell Driving e-research Across the Pacific `09 Sydney: 11/12/09

2 Current State of Cloud Utilization beginnings of adoption in research, education, gov support from Feds Federal CIO Vivek Kundra promoting aggressively (procurement portal) vendors adjusting services to reach Federal market Google federal cloud + FISMA in 2010 my opinion: many IT functions for.gov,.edu will be sourced as cloud services in 2-5 years good for researchers good for the planet exhilarating / unsettling for IT staff challenging architectural consequences

3 Selected Cloud Projects at Berkeley Lab Magellan project $32M DOE collaboration (LBNL-NERSC / ANL / ESnet) to explore feasibility of science cloud, with 100GE network collaborative and instructional services SaaS procurements (for recruitment, project management, ticketing, photo archive, and nipping at the heels of ERP) software development on AppEngine, SalesForce science pilots BLAST on Amazon Hadoop climate analysis on MS Azure Supernova Factory on Amazon EC2 cloud benchmarking for HPC

4 Benchmarks on Amazon EC2 Codes Science Area Algorithm Space Configuration Slowdown Reduction factor (SSP) Relative to Franklin CAM Climate (BER) Navier Stokes CFD 200 processors Standard IPCC5 D Mesh resolution Could not complete 240 proc run due to transient node failures. Some I/O and small messages MILC Lattice Gauge Physics (NP) Conjugate gradient, sparse matrix; FFT Weak scaled: 14 4 lattice on 8, 32, 64, 128, and 256 processors Erratic execution time IMPACT T Accelera tor Physics (HEP) PIC, FFT component 64 processors, 64x128x128 grid and 4M particles PIC portion performs well, but 3D FFT poor due to small message size MAESTRO Astroph ysics (HEP) Low Mach Hydro; block structuredgrid multiphysics 128 processors for 128^3 computational mesh Small messages and all reduce for implicit solve. source for slide and data: Lavanya Ramakrishnan, LBNL CRD 4

5 Benchmarking on Amazon EC2 test run performance >10x worse slow down factor using 32 cores on 4 nodes code is latency-sensitive also ran VASP code (modeling single-molecule junctions in nanostructures) with similar slow-down

6 Benchmarking on Amazon EC2 test run slow down factor using all cores on 8-core node code is not latency sensitive; 2-hour runs slow-down varies by application (we tested more)

7 Stepping Back: Implications for Infrastructure Providers? cloud-sourcing raises criticality of network for bulk data transfer for access to increasing number of external services , calendar, productivity, business apps network services (DNS?) and sufficient bandwidth is not the only requirement resiliency, fault tolerance peering arrangements with major cloud providers monitoring / management / fault isolation circuits not out of the question

8 Other Implications identity management & federation each SaaS procurement raises the issue variety of data management issues are data sets collocated with computing resources? performance? confidentiality? integrity? integration with existing workflows? cost analysis will be tricky to calculate cloud costs accurately, must estimate bandwidth, storage, CPU consumption & other metered services of course, total costs not well-understood in conventional architectures problem of infrastructure costs

9 Harder Problem: Cultural Shift for IT many researchers, end users will transition easily some IT professionals will not less demand for those close to the metal inevitable pressure to move up the application stack possible outcome: IT groups shrink better outcome: IT groups recover effort for innovation, integration, direct mission support how to maximize the latter possibility? managing expectations cloud-services can create space for local innovation not likely if cloud-sourcing is framed simply in terms of cost keenest challenges likely non-technical

10 Thank you. Greg Bell,