How to Do/Evaluate Cloud Computing Research. Young Choon Lee

Transcription

1 How to Do/Evaluate Cloud Computing Research Young Choon Lee

2 Cloud Computing Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. (NIST definition)

3 Cloud Computing Broadly falls into distributed computing Relevant/similar concepts Internet: largely information sharing Grid computing: primarily used as scientific platforms On-demand/elasticity + Utility computing

4 Cloud Computing Service Models IaaS (Infrastructure as a Service) AWS, Microsoft Azure, Google Compute Engine PaaS (Platform as a Service) Rackspace, Microsoft Azure SaaS (Software as a Service) Salesforce.com Deployment Models Private cloud Public cloud Hybrid cloud Community cloud

5 Clouds A (IaaS) cloud is basically a data centre Major public clouds host hundreds of thousands of servers, or even millions of them A single data centre often consists of thousands of servers or more; and they are virtualized

6 Example application Cycle Computing s cloud computing deployment 16,788 Amazon EC2 instance cluster with 156,314 cores across 8 regions in five continents for materials science experiments that collectively screen some 205,000 candidate molecules. This deployment has powered 2.3 million computing hours at a cost of only $33,000 which is equivalent to $68 million worth of equipment.

7 Problems Performance High performance/throughput Performance isolation and resource contention due to multi-tenancy Scalability Load balancing Cost Efficiency Resource failure Fault tolerance Failure recovery

8 Research Process Ideas / Feedback Design Analysis Implementation Experimental Evaluation

9 Research Process: ideas Pick up a promising/interesting domain, E.g., Cloud computing Big data In scientific discovery, the first three paradigms were experimental, theoretical and computational science (simulation), The Fourth Paradigm: Data-Intensive Scientific Discovery, Microsoft Research. Energy efficiency Survey/review literature Read papers, blog posts and even news paper articles Watch leading RGs and community/industry activities Find a real(istic) problem

10 Research Process: Design Algorithms, e.g., Scheduling Resource allocation Load balancing Fault tolerance Systems Hardware: new systems architecture, e.g., FAWN Software Res. management systems, e.g., Mesos, Workflow execution systems, e.g., DEWE

11 Research Process: Implementation A realization of the design Often involves Prototyping, e.g., Hadoop schedulers Software development (programming) Extensibility Portability Scalability Etc.

12 Research Process: Evaluation Experimental evaluation Simulations Only when you can WELL justify, e.g., energy efficiency with realistic data like Google cluster data, SPEC benchmarks Use existing simulators, e.g., NS-3, CloudSim Or write your own and make it open source Experiments in real systems Open-source systems, e.g., Hadoop, Xen, Docker Real systems like Amazon EC2 (AWS Education Grants)

13 Research Process: Analysis Cleaning and organising experimental results Scrutinizing results Conducting comparison study Describing results with effective use of figures and tables; colours, patterns and different chart types

14 Research Process: Analysis 100 CPU utilization m4r6m6r8m8r SM LRS 4m4r6m6r8m8r SM LRS 4m4r6m6r8m8r SM LRS 4m4r6m6r8m8r SM LRS 4m4r6m6r8m8r SM LRS 4m4r6m6r8m8r SM LRS exec idle iowait system user

15 Issues with Evaluation The seven deadly sins of cloud computing research, HotCloud sin, n. common simplification or shortcut employed by researchers; may present threat to scientific integrity and practical applicability or research Systems Benchmarking Crimes, Gernot Heiser, NICTA/UNSW. When reviewing systems papers (and sometimes even when reading published papers) I frequently come across highly misleading use of benchmarks. I call such cases benchmarking crimes. Not because you can go to jail for them (but probably should?) but because they undermine the integrity of the scientific process.

16 Issues with Evaluation: The seven deadly sins of cloud computing research Sin 1: Unnecessary distributed parallelism Sin 2: Assuming performance homogeneity Sin 3: Picking the low-hanging fruit Sin 4: Forcing the abstraction Sin 5: Unrepresentative workloads Sin 6: Assuming perfect elasticity Sin 7: Ignoring fault tolerance

17 Issues with Evaluation: Systems Benchmarking Crimes Pretending micro-benchmarks represent overall performance Benchmark sub-setting without strong justification Selective data set hiding deficiencies Throughput degraded by x% overhead is x% 6% 13% overhead is a 7% increase Same dataset for calibration and validation No indication of significance of data Benchmarking of simplified simulated system Inappropriate and misleading benchmarks Relative numbers only No proper baseline Only evaluate against yourself Unfair benchmarking of competitors Arithmetic mean for averaging across benchmark scores