Systematic Literature Review and Survey on High Performance Computing in Cloud

Transcription

1 Master s Thesis Electrical Engineering September 2012 Systematic Literature Review and Survey on High Performance Computing in Cloud Karthik Paladugu Sumanth Mukka School of Computing Blekinge Institute of Technology SE Karlskrona Sweden

2 This thesis is submitted to the School of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Electrical Engineering. The thesis is equivalent to 20 weeks of full time studies Contact Information: Author 1: Karthik Paladugu Author 2: Sumanth Mukka University advisor: Prof. Lars Lundberg Blekinge Institute of Technology School of Computing School of Computing Blekinge Institute of Technology SE Karlskrona Sweden Internet : Phone : Fax : ii

3 ABSTRACT Scientific and engineering applications require large number of calculations, to solve these calculations we need more number of processors, shared memory and multiple disks. Supercomputers are capable to process huge amount of calculations. In supercomputer, all of its power is utilized to execute few programs as efficient as possible. Later High Performance Computing (HPC) came into picture to solve advanced computational tasks of various applications in various fields. HPC uses the technology of supercomputer and computer clusters to perform advanced calculations. Now a days, HPC in Cloud came into existence to solve some of the technical and enterprise problems, which occur in traditional High Performance Computing and thereby increasing the potentiality of Cloud Computing. Developing HPC in cloud is one of the challenges of cloud developers and vendors. This paper focuses on documenting some of the research which is already done in the field of traditional HPC applications, HPC applications running in cloud and also their cost and performance aspects by using Systematic Literature Review. Thereafter, we have conducted surveys with few research questionnaires related to HPC in Cloud. Results From systematic literature review we found six major Domains of HPC applications, ten performance measuring tools and techniques, six major security issues. From survey results we found that Cloud can be used for running HPC applications when computational power is spiky and for loosely coupled applications. When utilization is more than 30% it is better to use traditional HPC. From survey & Interactive conversation different security solutions are obtained. From survey result we also got some performance measuring tools and techniques for HPC in cloud, they are HPC benchmarks, HPCC benchmarks, Ganglia, NPB etc. Conclusion From the information attained so far regarding whether cloud is ready for running HPC applications or not, we observed that cloud is not a viable solution for few applications but it is catching up fast. Cost aspects of running HPC applications in cloud, performance measuring tools and techniques, security solutions for HPC applications in Cloud are identified through Industrial experts by using Survey and Interactive conversation. Information gathered through surveys is compared with SLR result and is almost equal. Keywords: Benchmarks, Cloud Computing, High Performance Computing, Systematic literature review.

4 ACKNOWLEDGEMENT First and foremost we offer our sincere gratitude to our supervisor, Prof. Lars Lundberg, who has supported us throughout the thesis with his patience and knowledge. We attribute the level of our Master s degree to his encouragement and effort and without him this thesis would not have been completed or written. One simply could not wish for a better or friendlier supervisor. We would like to thank our survey participants who have contributed towards survey part of this thesis. Finally, we would also like to thank our loved parents for supporting us both morally and financially. Without their encouragement and motivation we could not able to complete this project. We would also like to thank our friends who helped us with their valuable suggestions and support. Karthik Paladugu, Sumanth Mukka ii

5 Table of Contents ABSTRACT...I ACKNOWLEDGEMENT... II TABLE OF CONTENTS... III LIST OF TABLE... V LIST OF FIGURE... VI 1 INTRODUCTION AIMS AND OBJECTIVES RESEARCH QUESTIONS BACK GROUND RESEARCH METHODOLOGY TYPES OF LITERATURE REVIEWS Systematic literature review (SLR) Systematic review process Search string Study Selection Criteria SLR process Tollgate approach Study Quality Assessment SURVEY PROCESS What is survey? Survey research Survey administration Flow of survey process INTERACTIVE CONVERSATION RESULTS SLR RESULTS Research question Research question Research question Research question SURVEY RESULTS iii

6 4.2.1 Survey question: Survey question: Survey question: Survey question: DISCUSSION VALIDITY THREATS Internal validity External validity Conclusion validity CONCLUSION RQ RQ RQ RQ FUTURE WORK REFERENCE APPENDIX A APPENDIX B APPENDIX C APPENDIX D APPENDIX E iv

7 LIST OF TABLE Table 1-1 Research questions & Methodology... 3 Table 3-1 Stages and their criteria involved in planning the review... 8 Table 3-2 Research Questions and Keywords... 9 Table 3-3 Inclusion Criteria Table 3-4 Selection Criteria Table 4-1 Hardware/Software Configurations Table 4-2 Cost and performance aspects Table 4-3 HPC applications in cloud Table 4-4 Security issues in HPC in cloud Table 4-5 Performance measuring tools and techniques for HPC in cloud Table 5-1 Overall SLR process v

8 LIST OF FIGURE Figure 3-1 Systematic review process... 8 Figure 3-2 Steps for SLR process Figure 3-3 Steps for selecting SLR papers using Tollgate approach Figure 3-4 Steps involved in selection of papers for RQ Figure 3-5 Steps involved in selection of papers for RQ Figure 3-6 Steps involved in selection of papers for RQ Figure 3-7 Steps involved in selection of papers for RQ Figure 3-8 Survey process Figure 6-1 Utilization of resources for Traditional HPC vs. HPC in Cloud vi

9 List of Abbreviations HPC SLR HPCC NPB IaaS PaaS EC2 DOE PFLOPS GFLOPS LHC BLAST YCSB HPL High Performance Computing Systematic Literature Review HPC Challenge NAS Parallel Benchmark Infrastructure as a service Platform as a service Elastic Compute Cloud Department of Energy Peta Floating-Point Operations Per Second Giga Floating-Point Operations Per Second Large Hadron Collider Basic Local Alignment Search Yahoo! Cloud Serving Benchmark High Performance Linpack vii

10 1 INTRODUCTION Cloud Computing allows customers to utilize resources and software s which are hosted by service providers. It mainly reduces infrastructure investment and maintenance cost. Computing infrastructure is not known to the users and resources are provided virtually in cloud. High Performance Computing (HPC) allows scientists and engineers to solve scientific, engineering and business problems using different applications that require high computational capabilities. [7] HPC applications are mainly categorized into two types; they are closely coupled applications, loosely coupled applications. To run these HPC applications it requires expensive hardware, and users have to wait to make use of the shared memory. We can run these HPC applications in Cloud; within the cloud no waiting time is required and there will be no need of Initial investment cost. [5]With the advent of Cloud Computing and benefits of IaaS and PaaS, scientists and engineers are able to deploy their HPC applications in the cloud without worrying about the costs associated with infrastructure and other costs. There are different applications which are running in cloud like Molecular modeling, Genome analysis, oil, Gas, Computer science etc. Cloud computing is attractive to scientists in the small research groups in computational science and engineering field. By using Cloud Computing these groups perform high performance calculations. When cloud is used for these HPC applications, the cost for running that application should be known. By analyzing this cost, research groups or vendors can know the return of investment and this cost estimation can determine which applications can use cloud. Scientists of small research groups should analyze the cost for running their application in cloud. They should know whether to run a part of application or complete application in Cloud to save money and resources i.e. in terms of cost, performance and infrastructure. [13] Scientists should use their known performance characteristics (e.g., transferred data, execution time) of their applications to analyze and estimate cost. Some of the cost models are provided by cloud providers in their pricing specification. Basic cost models that are provided by cloud providers in pricing specification are Mds (Data storage), Mcm (Computational machine), Mdfi (Data transfer into the cloud), and Mdfo (Data transfer out of the cloud). [15] In 2008, Amazon has begun offering Cloud Computing services through their Elastic Compute Cloud (EC2) product. With this product users can rent virtual servers in hourly basis and can dynamically change the server requirements as load fluctuates over time. This flexibility primarily driven to the user s interest in Cloud Computing. From the year 2008 and 2009 Amazon started offering array of different EC2 cluster instances. Amazon EC2 is 1

11 the one of the most powerful public cloud service provider in the market. [14]Public clouds brought High Performance Computing resources nearer to ordinary companies or small scale companies. For temporary projects, cloud solutions are less costly for running HPC applications than using normal traditional HPC. Due to high cost, maintenance and large power consumption, traditional HPC vendors haven t built large HPC clusters. For small scale companies, running their HPC applications in cloud is better. That HPC cluster should have the capability to run multiple HPC applications simultaneously. Earlier it took months to run applications on local hardware and now due to usage of HPC clusters in cloud, it is taking hours to complete the work. With this, companies not only decrease their costs, but can also serve their customers in better and faster way. The main intention of this research is to know whether cloud is a viable solution for running HPC applications or not. Systematic literature review, survey and Interactive conversation have been conducted for our research. Different HPC applications in Cloud, Security issues, and different performance measuring tools and techniques are gathered from the empirical data by using Systematic Literature Review, and then the gathered empirical data from SLR is compared with results obtained from Survey and Interactive conversation. 1.1 Aims and Objectives Aim The main aim of this research is to know whether High Performance Computing is suitable for Cloud or not and also to know which traditional HPC applications have solved in cloud. Objectives To study different kinds of traditional HPC applications. To study which traditional HPC applications have been solved by HPC in Cloud. To study various security issues involved in HPC in Cloud. To know Cost and performance aspects of HPC applications in Cloud. 2

12 1.2 Research questions Table 1-1 Research questions & Methodology SNO # RESEARCH QUESTIONS RESEARCH METHODOLOGY 1. What are the cost and performance aspects of HPC Systematic Literature Review, applications in the Cloud? Survey, Interactive conversation 2. Which HPC applications have been executed in the Cloud? Systematic Literature Review 3. What type of security issues are involved in HPC in Cloud? Systematic Literature Review and Survey, Interactive conversation 4. What are the different kinds of tools and techniques that are used to measure the performance of HPC in Cloud? Systematic Literature Review and Survey, Interactive conversation 3

13 2 BACK GROUND Supercomputer is used to carry out high-speed calculations as their processing capacity is very high. High Performance Computing is similar to supercomputer, but it has added advantage of computer clusters. HPC is used to solve advanced calculations in the field of science and engineering. All the applications which are solved by supercomputers can be solved by HPC. Scientists, engineers and analysts in the leading companies and research stations are depending on HPC to solve challenging problems [2] in the fields like engineering, finance, revenue management, manufacturing, risk analysis, life and earth sciences etc. All these problems can be solved by HPC and mainly it should have cost control. Cloud Computing has the ability to effectively reduce cost and complexity. In Cloud Computing large amount of data and computing resources can be accessed remotely over the internet using personal computer or other devices. Cost control and many other concerns like speed of execution and remote access leads to HPC development in the cloud. [1] Case study related to importance of Amazon EC2 in HPC is provided in terms of cost and performance. [2] IBM HPC cloud s importance is explained for both private and public cloud. Authors explained about HPC management suite provided by IBM and mentioned the need of HPC to turn into cloud. [3] Need of Supercomputers and their history have been discussed. Authors also mentioned how to compare the speeds of different Supercomputers. [4] Methods used for attaining energy efficiency in cloud are mentioned. Energy efficiency is attained by using energy efficiency Hardware and energy efficiency scheduling techniques are mentioned. [5] Authors focused on documenting some of research already done in the field of HPC applications and their current state in Cloud. [6] Smart irrigation system is described, here irrigation system uses supercomputer to control and communicate. This white paper [7] discusses about renting of compute resources and this also discusses about importance of HPC cluster in cloud of Penguin Company. [8] Authors discussed about whether to choose Linux or Windows for High Performance Computing and also used different benchmarks to do performance analysis. In this article [9] author discusses about hybrid cloud and use of hybrid cloud compared to private and public clouds and also authors mainly concentrated on enterprise computer users. [10] Supercomputing application which was launched by NASA is described and this application is used to analyze large earth science Data sets. [11] Importance of cloud for HPC applications depending upon application characteristics such as network performance, communication volume and cost trade-offs is considered and evaluated. [12] Goal of 4

14 Magellan project was to investigate the important role of Cloud Computing in addressing the computing needs for DOE of science. 5

15 3 RESEARCH METHODOLOGY We have used Systematic Literature Review (SLR), Survey and Interactive conversation as our research methodologies. Main aim of systematic literature is to provide comprehensive summery of literature related to a research question. SLR is conducted by using search strings, Search strings are formed by using keywords and we have used PICO method to define keywords. The obtained search strings are used in different databases and found few research papers related to research questions. [16] New interpretation of old material (i.e. results obtained from SLR) is used in this paper. Mainly Literature review discusses published information in a particular subject area. Survey is conducted by framing some questionnaires and posing them to the experts in the related field. Here our task is to identify different groups, practitioners and different organizations working in the area of Cloud Computing and HPC. Questionnaires are prepared and are posted in the form of polls in professional websites like LinkedIn (Interactive conversation). We identified practitioners from the organizations who are working with cloud and HPC. Survey process is conducted by sending survey link to practitioners through s. 3.1 Types of literature reviews There are different types of literature review like traditional review, systematic literature review. Traditional review will give the summary of all the study done on that particular field, it do not have any systematic steps to follow by which the output is not so advisable to use as a background for any study. Whereas systematic literature review is a scientific approach to identify, evaluate and interpret the available research related to specific research questions. It uses a well-defined method by which a user can rely on the outputs of the research question. We will use systematic literature review to search and collect research content or information related to traditional HPC & HPC in cloud. By using this kind of methodology we will get fair results. 6

16 3.1.1 Systematic literature review (SLR) [16] A systematic literature review is a means of Identifying, Evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomena of interest (Kitchenham, 2004). Reasons for performing SLR o It can identify gaps in current research for further investigation. o It can be used as framework for new research activities. o To generate a new hypothesis. Importance of systematic literature review o Literature review has less scientific value. o Systematic literature review approach is fair. o It summarises all existing information about some phenomenon in a fair and unbiased manner. Advantages o Methodology is well defined and due to this results of literature will have less repeatability. o New research activities can be developed. Disadvantages o More effort is required Features of SLR o Systematic review starts by defining review protocol. o Based on search strategy systematic review is defined. o Search strategy is documented. o Quality criteria for paper selection of primary studies are considered. o Systematic review can be used as a prerequisite for quantitative Meta-analysis Systematic review process [16] A systematic literature review involves several discrete activities. They have been divided into three main phases. Planning the review Conducting review Reporting review 7

17 Figure 3-1 Systematic review process Planning the review Stages involved in planning review Table 3-1 Stages and their criteria involved in planning the review SNO# Stages Criteria 1. Identification of the need of research Research gap Fair research 2. Specify research question Cost aspects of HPC Security issues Performance related aspects 3. Developing review protocol Research questions Choosing appropriate Keywords Developing the search strings. Primary study selection Development of review protocol After identifying the need of research, we prepared research questions and the review protocol is designed. Review protocol defines specific procedures for conducting the systematic review process. By using this procedure we can attain fair and unbiased information. This protocol development has different stages, such as search strategy, selection criteria, quality assessment criteria, data extraction form and data synthesis strategy Search strategy This strategy helps in answering research questions effectively. By using keywords we have developed search strings. Search strings are constructed by identifying Synonyms and alternative spellings for each of the question elements and link them by using the Boolean OR and Boolean AND. 8

18 Authors have defined keywords by using PICO method [16]; keywords which are defined from PICO method are used to construct search strings. PICO: Population Intervention Comparison Outcomes Population: The population might be any of the specific role, application and area. Population- Cloud Computing Intervention: The intervention is the tool or technology or procedure that addresses a specific issue. Intervention- High Performance Computing Comparison: This is a tool or technology or procedure with which intervention is being compared. comparison- Traditional High Performance Computing Outcomes: outcomes should relate to factors of importance to practitioners such as improved reliability, reduced production costs, and reduced time to market. All outcomes should be specified. Outcomes- cost aspects, performance aspects, applications, tools and techniques Table 3-2 Research Questions and Keywords SNO Research Question Keywords RQ.1. What are the cost and performance aspects of HPC applications in the Cloud? Cost, Performance, HPC, Application, Cloud. RQ.2. Which HPC applications have been executed in the Cloud? HPC, Application, Cloud, executed. RQ.3. What type of security issues are involved in HPC in Cloud? Security, HPC, Cloud, RQ.4. What are the different kinds of tools and techniques that are used to measure the performance of HPC in Cloud? Tools, Performance, HPC, Cloud 9

19 3.1.3 Search string Following are the search strings that are appropriately designed by using keywords, keywords are derived from research questions by using PICO method. These search strings are constructed by using Boolean ANDs and ORs. Search string for Research question 1 (HPC applications OR High performance computing applications OR scientific applications) AND (Cloud computing OR on demand computing) AND (Cost OR Monetary OR price OR Economics) AND (Performance OR quality) Search string for Research question 2 ( HPC OR hpc applications OR high performance computing applications OR scientific applications) AND (cloud computing OR on demand computing) AND (executed OR solved OR shifted OR moved OR processing) Search string for Research question 3 ( HPC OR hpc applications OR high performance computing applications OR scientific applications) AND (cloud computing OR on demand computing) AND (Security OR issues OR problems OR Threats) Search string for Research question 4 ( HPC OR hpc applications OR high performance computing applications OR scientific applications) AND (cloud computing OR on demand computing) AND (tools OR benchmark) AND (performance) Resources Search strings are used in Digital libraries for getting related research content. The articles, journals, conference papers, and workshop papers will be identified from the most authentic Digital databases that are scientifically and technically peer reviewed such as :- Inspec Compendex IEEE Xplore ACM Digital Library Springer Link Science Direct 10

20 3.1.4 Study Selection Criteria Study selection criteria was used to determine which studies are included in, or excluded from systematic review. Main intention of this criterion is to identify those primary studies that provide direct evidence about research question. Following table shows Inclusion criteria for research Inclusion criteria Table 3-3 Inclusion Criteria SNO# Stage Criteria 1. Overall Language: English 2. Title, Keywords and abstract 3. Introduction and Conclusion Paper published in journal/conference/workshop/ web articles. Date of publish Non-duplicate Based on keywords and search strings. Based on the content which matches to our research questions. Contains HPC in cloud background. Mainly focuses on answering our research questions. 4. Full text Presence of the empirical data in the paper. Main focus on the benefits, challenges and tools to the performance of running High performance computing applications in cloud Exclusion Criteria The research articles will be excluded that do not meet the criteria mentioned in above table 11

21 3.1.5 SLR process Figure 3-2 Steps for SLR process 12

22 3.1.6 Tollgate approach Figure 3-3 Steps for selecting SLR papers using Tollgate approach 13

23 In this stage both researchers participate individually, papers from different databases are considered. This scenario involves total three stages: Stage 1: Here, at this stage search strings are used for paper selection Papers between are considered and also a language criterion is also included. Stage 2: Here, at this stage both researchers consider Title and Abstract for paper selection, related papers are considered and remaining papers are excluded. Stage 3: Here, at this stage both researchers consider papers from stage 2, for paper selection in this stage they consider Introduction & Conclusion as criteria. Related papers are considered and remaining papers are excluded. Final stage: Here, at this stage full text discussion is the selection criteria, after discussion related papers are selected Steps for final selection of papers From our structured research questions, we have constructed search strings to use in Electronic databases. Search strings are constructed by identifying Synonyms and alternative spellings for each of the question elements and linked them using the Boolean OR and Boolean AND. Later in an attempt to perform exhaustive search we have identified four Databases. Engineering village Springer link ACM digital Library Science direct Screening By search: By using Strings, we started our search in engineering village. String is placed in advanced search box and search has been performed. After obtaining few papers, we applied Inclusion and Exclusion criteria and few papers are filtered. Later based on keywords remaining papers are filtered. Same above process is repeated for Science Direct and papers are filtered. Finally all the papers obtained from Engineering Village and Science direct are combined. 14

24 Filtering by duplicates: Duplicate papers are removed by comparing the papers obtained from both databases. Screening by Abstract, Introduction & Conclusion: In this step both researchers filter the papers based on Abstract, Introduction & conclusion criteria. Screening by Full Text: Here researchers filter the papers based on full text. Finally filtered papers are review by authors and they are used as preliminary materials in our study. 15

25 RQ 1: What are the cost and performance aspects of HPC applications in the Cloud? Screening By search Manual search by Keywords and title Combining all databases 137 Filtering by Duplicates Screening by abstract, Introduction conclusion Screening By Full text 12 Figure 3-4 Steps involved in selection of papers for RQ 1 16

26 RQ 2: Which HPC applications have been executed in the Cloud? Screening By search Manual search by Keywords and title Combining all databases 301 Filtering by Duplicates Screening by abstract, Introduction conclusion Screening By Full text 10 Figure 3-5 Steps involved in selection of papers for RQ 2 17

27 RQ 3: What type of security issues are involved in HPC in Cloud? Screening By search Manual search by Keywords and title Combining all databases 151 Filtering by Duplicates 76 Screening by abstract, Introduction conclusion 28 Screening By Full text 11 Figure 3-6 Steps involved in selection of papers for RQ 3 18

28 RQ 4: What are the different kinds of tools and techniques that are used to measure the performance of HPC in Cloud? Screening By search Manual search by Keywords and title Combining all databases 147 Filtering by Duplicates 83 Screening by abstract, Introduction conclusion 32 Screening 13 By Full text Figure 3-7 Steps involved in selection of papers for RQ 4 19

29 3.1.7 Study Quality Assessment We have developed this quality assessment checklist to assess the Individual studies. If the study fulfills assessment criteria then it is filled with yes. [16] Quality assessment checklist is prepared based on kitchenham. Quality assessment provides detailed inclusion/exclusion criteria. Quality assessment checklist provides recommendations for further research. Table 3-4 Selection Criteria SNO# Quality Assessment Checklist Yes/No 1. Does the research paper clearly specify the research methodology? Yes 2. Does the research methodology appropriate for problem under concern? Yes 3. Are the results of study properly mentioned? Yes 3.2 Survey process What is survey? [17] It is a method used for gathering information from a sample of individuals. Figure 3-8 Survey process 20

30 3.2.2 Survey research Survey research is a commonly used method of collecting information about population of interest. There are different types of surveys and many methods of sampling. 1. Type of survey used: Questionnaires 2. Sampling method used: stratified Sampling Questionnaires [18] Predefined series of questions are used to collect information from individuals. o Two most types of survey questions are open-ended and closed ended. o We used open-ended questions for our survey process Sampling [18] It is a technique in which subgroup of population is selected to answer the survey questions. There are different methods of sampling methods such as 1. Simple random sampling 2. Cluster sampling 3. Stratified sampling 4. Non random sampling We are using stratified sampling method as our sampling technique for survey. In this sampling researcher first identifies the people in the population who have desired characteristics related to research. Here we considered experts who are particularly working in the field of Cloud Computing, HPC Survey administration [18] Survey can be administrated in three ways: 1. Through mail 2. By telephone 3. Face-to-Face Flow of survey process Different stages involved in our survey process are: 1. Plan and Development 2. Survey Design 3. Implementation of survey 21

31 4. Collection of Results 5. Analysis of data Plan and Development stage In this stage we have collected background data for designing survey i.e. for questionnaire preparation, we also prepared outline of questionnaires and outline of sampling process (i.e. Groups to selected, experts). In this stage, Planning of survey administration on questionnaires is performed. Outline of survey process is also done in this stage. Survey design In this stage authors have divided Survey design into two parts: 1. Demographic Information: consists of Closed ended questions 2. Contextual section: consists of open ended questions Final survey questionnaires are prepared. Revision of questionnaires and Preliminary test of designed survey is performed by sending the survey link to our supervisor. Implementation of survey In this stage survey link is sent to selected practitioners, focussed groups. Mainly finalization of survey administration is considered here. For demographic information authors used closed ended questions (work experience etc.) and contextual section consists of open ended survey questionnaires. Survey is conducted by sending survey link to the mails of selected practitioners and also through face-face interview. Collection of results By using we have designed our survey and we have collected results. In this stage confidentiality is provided for practitioner s information in survey. We have sent our survey link to cloud & HPC related practitioners, posted link into cloud groups in professional websites like LinkedIn. Seventeen practitioners participated in our survey. Analysis of Data Finally in this stage verification and validation of results are performed. Authors verified that whether all the practitioners have provided answers for all questionnaires. Seventeen practitioners were selected finally for all results. 22

32 3.3 Interactive conversation We have also used interactive conversation survey method for conducting survey. Here in this method, we have used professional websites like LinkedIn, different blogs related to Cloud and HPC for conducting survey. We have posted our questionnaires into those sites and had an interactive conversation regarding our research. We had conversation with few organizations Tech support (i.e. using live chat) who is working in the field of Cloud and HPC. 23

33 4 RESULTS 4.1 SLR Results Research question 1 What are the cost and performance aspects of HPC applications in the Cloud? Cost model: With the reference of paper [32] cost analysis has been done to know whether clouds are ready for running HPC applications compared to Supercomputers. As a part of comparison we considered NASA S primary supercomputer Pleiades and Amazon HPC cc1.4xlarge instance [32]. Pleiades supercomputer consists of total 11,776 nodes, 126,720 cores and its peak performance is 1.75 PFLOPS. We assumed the entire system cost and maintenance cost is $9 million per year. Overall both systems environment is mentioned in below table. Table 4-1 Hardware/Software Configurations Pleiades Amazon HPC CPU Intel Xeon X5570 Nehalem Intel Xeon X5570 Nehalem Clock Speed 2.93 GHz 2.93 GHz No. of cores/node 8 8 RAM 24 GB 23 GB Hyper-threading Enabled Enabled Network Technology InfiniBand 4XQDR 10 GigE Interconnect Topology Hypercube Unknown OS Type SLES11SP1 SLES11SP1 Virtualization No Yes Multi-tenancy No No Compiler Intel Intel MPI Library MPT 2.04 Open MPI File Structure Lustre NFS 24

34 We have calculated cost by considering 64 cc1.4xlarge instances of Amazon Ec2. This quadruple extra-large instance is mainly considered because it is used for running HPC applications [32]. Cost of this instance running with SUSE Linux server is $1.4 per hour. [71] At an average of 30.5 days in a month, the instance runs 732 hours per month. For 64 instances cost will be $ per month. Cost calculation is based on 100% utilization of resources i.e. full utilization of instances. Cost for 64 instances per year is $0.78 million. [32] A study was conducted to compare the performance of amazon EC2 HPC instances and Pleiades NASA s largest supercomputer, in their study authors used a suite of benchmarks ranging from low level kernels to full applications for performance comparisons between Pleiades and Amazon Ec2. From HPCC benchmark results on 64-cores [32], performance of Pleiades is Gflops and Amazon Ec2 is Gflops. From the performance results we can conclude that Amazon Ec2 HPC performance lags Pleiades supercomputer for most of NASA s HPC applications. Above cost estimate shows that cloud users will have monetary gain. Before running HPC applications in cloud, cloud practitioners/users have to estimate the cost for running particular application in cloud (i.e. Amazon Ec2, Microsoft Azure etc.). To estimate cost we have to consider the following. 1) Compute Clock Hours of Server Time Machine Configuration Machine Purchase Type Number of Instances 2) Storage Additional Storage Backups Data Transfers 3) Load Balancing 4) Detailed Monitoring 5) Auto Scaling 6) Elastic IP Addresses 7) Operating Systems and Software Packages 25

35 Table 4-2 Cost and performance aspects Reference Cost and performance aspects Number [21] Scalability & reliability is achieved for Geoscience application in Cloud. Authors found that the standard large Linux cluster should be used to balance cost & performance (it only costs $0.34). [22] Selecting an instance type which suits your application can give significant time and monetary advantages. Authors suggested that loosely coupled scientific applications can be implemented on Cloud. [23] Cloud provides individual users with the capability to customize their cluster instances. Physical cluster: increased Cluster utilization lowers actual cost but produces longer queue waiting time and also higher performance variability. Elasticity of on demand public clouds will reduce the above problem occurred in Physical cluster. [11] Cloud can be viable platform for some HPC applications depending upon application characteristics such as communication volume and sensitivity to OS noise. Cloud is better suitable for low communication intensive applications such as embarrassingly parallel. [25] There is a strong correlation between the percentage of time an application spends communicating, and its overall performance on EC2. Depends on the communication pattern of application can have impact on performance. [26] By providing right amount of storage and compute resources cost can be reduced with no impact on application. [27] Amazon EC2 can be used when low utilization rates are present for a scientific application. [28] Depending on the type of virtual machine (VM) cost fluctuates. When number of virtual cores is increased, then speed of execution increases with small degradation of performance. 26

36 [29] Cloud Computing offers a powerful cost effective resource for scientists, especially for compute and memory intensive applications. Commercial clouds may not be better suitable for large scale applications. [30] Virtualization overhead present in EC2 results in performance degradation. Disable the instance once the application code is complete to avoid being charged for unused or ideal instance. [13] Basic cost models should be known, before transferring scientific applications into cloud. Authors of this paper provided some cost models Research question 2 Which HPC applications have been executed in the Cloud? Table 4-3 HPC applications in cloud Reference Main Domain Description Number [22] Biomedical applications In this paper authors have used two pleasingly parallel bio medical applications. Cap3 is used to test the assembly of genome fragments, GTM & MDS interpolation are dimension reduction algorithms and play key role in scientific data visualization.for data processing using of Map Reduce is better. [34] Atmosphere-ocean simulation By using coupled atmosphere ocean configuration of the MIT General Circulation Model (MITgcm) application on Amazon Ec2 authors concluded that s mall sized HPC applications are suitable in cloud. 27

37 [35] Physics Experiments By using physics experiments Dzero, LHC, SLAC Amazon S3 ability is evaluated interns of different parameters. [36] Seismograms(ground related information) Seismogram application which is used to find the location of earthquake and source parameters of earthquake. Seismograms requires large amount of computing process and storage. [26] Astronomy( montage) Authors used montage science application, it is a real life Astronomy application. Montage creates science-grade image mosaics from multiple input images.using montage example on A mazon EC2, they showed data-intensive application with small computation is cost effective. [38] Medical(Gene &fmri) Two case studies have been presented: the classification of gene expression data by using Aneka cloud deployed on amazon EC2 infrastructure and execution of scientific workflow application (fmri) on EC2. [39] Moderate Resolution Imaging Spectroradiometer This is a scientific application which is useful to many environmental studies. [40] Neuroscience It is an e-science project, in this project investigation of how brain encodes, transmits and processing information. 28

38 [21] Geoscience Weather research and forecasting GEOSS clearinghouse is a web based Geographic Metadata Catalog System. It manages millions of metadata of the spatially referenced resources for the Global Earth Observation. [24] This model is one of best models to carry out weather prediction. WRF demands large amount of CPU power. Authors proposed lightweight virtual machine mechanism in cloud for running WRF application. Results show that proposed mechanism is better to achieve good performance Research question 3 What type of security issues are involved in HPC in Cloud? Reference Number Table 4-4 Security issues in HPC in cloud Security issue Description [41] Data security This issue can be addressed by applying internet key exchange scheme. Due to low efficiency of above scheme, a new scheme called CCBKE is applied for scientific applications in cloud. [42] Stolen credentials Here, security incidents occurred by using stolen credentials at the national Centre for supercomputing applications. 29

39 [43],[44] Data management [46],[48] Loss of governance Data lock in Compliance etc. [45] Security measures for network virtualization (Inter cloud communication) [47] Multitenant related security issues Usages of Cloud Computing for medical data storage services are discussed and security issues involved in this process are also discussed. Tinyvine cloud platform is tested for inter cloud communication by using MPI benchmarks and BLAST application. Analyzing security issues related to multitenant and literature review of relevant threats and solutions to them are discussed. [49],[50],[51] Confidentiality & Integrity Verification of confidentiality & Integrity of data is big problem; this was addresses by using TCCP. This guarantees confidential execution of guest VM Research question 4 What are the different kinds of tools and techniques that are used to measure the performance of HPC in Cloud? Table 4-5 Performance measuring tools and techniques for HPC in cloud Reference Benchmark Use Description Number Name [52] 1.Synthetic Assess network Performance evaluation benchmark efficiency of Azure and nimbus cloud 2. Real life Evaluation of is done by using these benchmarks. applications computation power [53] DGSIM simulator Analyses performance of clouds Performance analysis in terms of many task computing (MTC) users takes place. 30

40 [54],[64] Macro & micro benchmarks Performance of clusters [55] YCSB Performance of cloud serving stores [56],[57], HPL Performance [59],[60] evaluation of cloud Examination of difference between CPU of amazon EC2 & traditional HPC. Four different databases performance is evaluated in terms of read, write. Use of cloud for scientific computing for cloud is investigated, to compare with Top 500. [57],[58], NPB( NAS Evaluate Case study on HPC applications in [59],[62] parallel performance of cloud is conducted. benchmark) parallel supercomputers [58] STREAM Memory bandwidth Explores utilization of EC2 for benchmark test scientific applications. [61],[58] IOR benchmark Performance of I/O characteristics [62] NETPERF Network benchmark performance To benchmark the results of I/O characteristics of Amazon, Magellan. In order to overcome network overhead problem, new mechanism called LCM was developed and it was evaluated. [25] Integrated Measures time taken Evaluation of performance of HPC performance by an application in applications in cloud environment monitoring tool MPI on task by task by using real world applications. basis 31

41 4.2 Survey results Survey question: 1 In which cases do you expect that the cost for running an HPC application in the cloud would be lower/higher than buying a super computer and run the application on that? You pay for what you use For running loosely coupled applications or some tightly coupled programs with less communication Consider Cost estimation between in-house cluster & cloud Computational power is spiky If utilization of supercomputer is more than 70% Survey question: 2 Can a public Cloud be used for High Performance Computing? Yes, public cloud can be used for HPC Survey question: 3 What are the different kinds of security solutions used by different Cloud vendors for HPC? For sensitive purpose applications, it is better not to run them in cloud and virtualization/hypervisors is a better solution for security issues. Encryption also gives better security solution for data management issues Survey question: 4 What are the different kinds of tools that are used to measure the performance of HPC in Cloud? All the standard tools which are used in local supercomputers can be used and also micro benchmarks can be used 32

42 5 DISCUSSION Table 5-1 Overall SLR process SNO# Stages Criteria 1. Research questions Research questions have been formed on the basis of verification of research gap. 2. Development of review protocol In this phase we want to use empirical related work 3. Validate review protocol Papers obtained from above criteria are reviewed 4. Primary studies Abstracts, Keywords, Conclusions are Main Criteria 5. Assess study quality We are using empirical data for our work i.e. reports which are done in an experimental way (Magellan report), white papers etc. The main goal of this research is to know whether cloud is better solution for running HPC applications. Here in our research we have identified previous works related to traditional HPC applications and High Performance Computing applications running in Cloud. For this research, we have followed SLR and survey process to answer research questions. Main motivation to use SLR process is, it is an evidence based approach. Overall process for conducting SLR is mentioned in Table 11. [70] SLR is a qualitative research method, in this method only particular category of population (research papers) is selected for given study. We have used snowball sampling method which is one of the most common sampling methods used in qualitative research. Snowball sampling is used to find hidden population i.e. if the sample (research content) for study is limited. For RQ1 we did SLR, survey and also designed one cost model to show whether cloud is better solution for running HPC applications, we compared Amazon HPC instance & Pleiades supercomputer (NASA) to analyze cost. For cost analysis we have selected similar execution environments of Pleiades supercomputer and Amazon HPC instances. For RQ2 from SLR we found six main domains of HPC applications. For RQ3 from SLR we found six major security issues and from survey we got some security solutions. For RQ4 from SLR we found 10 different kinds of tools and techniques to measure performance and from survey we received some tools. Almost all performance measuring tools and techniques used for traditional HPC can be used for HPC in Cloud. 33

43 We had an interactive conversation with few experts from organizations and from LinkedIn to get results for survey. We did our level best to get responses from many practitioners for our Survey. We were successful in getting responses from only seventeen members; we have excluded the responses which are inappropriate, we considered survey results from practitioners who have good experience in related field. We have sent our survey link to Experts who are working in organizations like SGI, UNIVA etc. The main reason behind not getting responses was due to fact that industry experts don t have enough time to fill it. In SLR results section we have tabulated each primary paper attained. In survey results section we have provided overview of results which we got from practitioners through conducting survey. We have validated our SLR results with survey results. Finally validate results are presented in Results section. 5.1 Validity Threats Internal validity [66] Internal validity is the approximate truth about evidences regarding cause-effect relationships. For both systematic literature review and survey internal validation is done SLR [66]Selection bias is one of internal validity threat, to mitigate this threat in our research we followed systematic method suggested by kitchenham. Search strings were formed from research questions and discussed with the supervisor. Search strings are used in different databases to find related research papers. Finally related research papers were selected by following some study selection criteria. To mitigate selection bias thereat we have selected representative research content i.e. which is related to our research. We have kept following selection criteria. Year : Keywords Title Abstract and conclusion 34

44 Survey Questions in survey can be misinterpreted by experts, which may affect the results. To mitigate this threat, we had a discussion with our supervisor related to our questionnaires and our supervisor suggested some improvements. We did the above process till the final approval of our supervisor External validity [67] External validity is the degree to which the conclusions in your study would hold for other persons in other places and at other times. There are three major threats in external validity because there are three ways researchers could be wrong i.e. places, peoples, times Survey There might be chances of getting low responses for survey, thereby to mitigate this threat we have conducted survey by using survey link and also we sent our survey link to some practitioners & organizations who are working in field of Cloud and HPC. To keep respondent s dropout s rate low we have also conducted our survey in LinkedIn Conclusion validity [68] Conclusion validity is the degree to which the conclusion we reach is credible or believable. Main threat to conclusion validity is a factor that can lead you to reach an incorrect conclusion about a relationship in your observations SLR There might be chances of missing relevant articles, to mitigate this threat study selection criteria and data extraction form was clearly defined by both researchers for primary studies. After getting primary papers we had a discussion regarding the differences in data which we have gathered. After thorough discussion we reached to an agreement on how to identify required data. Finally we gathered the information required for research questions from selected papers Survey Unbiased results are a threat in Survey, to mitigate this threat we have selected Experts related to particular Area of Field i.e. cloud & HPC, So there are less chances of getting 35

45 ambiguous results, after getting results they were shown to supervisor. After getting the approval from supervisor we compared SLR results with Survey. We finally gave conclusions for all research questions in conclusions section. 36

46 Utilization of resources 6 CONCLUSION 6.1 RQ1 From cost model analysis, SLR and Survey we can conclude that, if utilization of a traditional HPC is more than 30% then the monetary benefit will be limited for HPC in Cloud. From interactive conversations we can conclude that the cloud can be used for running HPC applications when low spiky loads are present. From the systematic literature review and surveys we can conclude: Scientific applications with less communication are best suited for HPC in Cloud. Scientific applications with more communication are best suited for traditional HPC. 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% MPI Software Architecture Multi threading HPC in cloud Traditional HPC Figure 6-1 Utilization of resources for Traditional HPC vs. HPC in Cloud 6.2 RQ2 From our SLR results we can conclude that HPC in cloud supports six main domains and many other applications. Six main domains are Computational Biology (BIO), Computational Chemistry and Materials (CCM), Computational Fluid Dynamics (CFD), Finite Element Analysis (FEA), Computational Electromagnetics (CEM) and Ontologies (ONT). Other applications include Physics experiments, Seismograms, Astronomy, and Neuroscience etc. 37

47 6.3 RQ3 From our SLR results we can conclude that HPC in Cloud has six major security issues, they are Data Security, Stolen Credentials, Data Management, Network Virtualization, and Multitenant related security issues, Confidentiality & Integrity. From the conducted survey we are presenting security solutions, they are 1. Use of Encryption techniques 2. It is better not to run Sensitive applications. 6.4 RQ4 From our SLR results and Survey we found twelve different kinds of tools and techniques to measure the performance of HPC applications in cloud. They are synthetic benchmark, real life applications, DGSIM Simulator, Macro & Micro benchmarks, YCSB, HPL, NPB (NAS parallel benchmark), STREAM benchmark, IOR benchmark, NETPERF benchmark, Integrated Performance Monitoring tool, HPCC benchmark etc. All the tools and techniques which are used in traditional HPC can also be used for HPC in Cloud. 38

48 7 FUTURE WORK In this study, we have focused on qualitative analysis of cost and performance aspects of HPC application. In future, quantitative analysis (experimentation) can be done by using different HPC applications on traditional HPC and HPC in Cloud to know the cost & performance, particularly by using different high and low data intensive scientific applications. 39

49 8 REFERENCE [1] S. Hazelhurst, "Scientific computing using virtual high-performance computing: A case study using the amazon elastic computing cloud," in SAICSIT 2008: riding the wave of technology Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists, Wilderness, South Africa, October 6, October 8, 2008, pp. vol. 338, pp [2] Smart IBM Solutions for High Performance Engineering Clouds, white paper, IBM, June, [3] Jan Matlis, A brief history of super computers, Computerworld, 31 May2005, [4] Berl, E. Gelenbe, M. Di Girolamo, G. Giuliani, H. De Meer, M. Q. Dang, and K. Pentikousis, "Energy-Efficient Cloud Computing," The Computer Journal, Published by Oxford University Press on behalf of The British Computer Society, vol. 53, no. 7, pp , August [5] Sindhu Mani Sanjay P. Ahuja, "The State of High Performance Computing in the Cloud," Journal of Emerging Trends in Computing and Information Sciences, vol. 3, no. ISSN , pp , February [6] Ingfei Chen, Smart Irrigation: A Supercomputer Waters the Lawn, July 4, 2011, [Online]. Available: [Accessed: September 2012]. [7] John S. Peterson, High Performance Computing with ClearCube PC Blades, Oct 2011, [Online]. Availible: [Accessed: September 2012]. [8] Choosing between Windows and Linux for High Performance computing, white paper, Crimson Consulting Group, January [9] Denoid Tucker Tucker Hybrid Cloud Ready for Takeoff in 2012, January 17, 2012, [Online]. Available: 17/hybrid_cloud_ready_for_takeoff_in_2012.html?featured=top [Accessed: September 2012] [10] J. Nicholas Hoover, NASA Launching Supercomputing Application, April 16, 2010, [Online]. Available: [Accessed: September 2012]. [11] A. Gupta and D. Milojicic. Evaluation of hpc applications on cloud, Technical Report HPL , HP Laboratories, [12] Katherine Yelick, Susan Coghlan, Brent Draney, Richard Shane Canon, The Magellan Report on Cloud Computing for Science, U.S. Department of Energy, USA, December, [13] H.-L. Truong and S. Dustdar, Composable cost estimation and monitoring for computational applications in cloud computing environments, in 10th International Conference on Computational Science 2010, ICCS 2010, May 31, June 2, 2010, Amsterdam, Netherlands, 2010, vol. 1, pp [14] Greg Elkinbard, HPC in the CLoud. Running compute intensive task in cloud, October 25, 2011, [Online]. Available: 25/running_compute_intensive_tasks_in_the_cloud.html [Accessed: September 2012]. [15] Amazon Web Services, Amazon Elastic Compute Cloud, [Online]. Available: [Accessed: September 2012]. [16] B. Kitchenham, S. Charters, "Guidelines for Performing Systematic Literature Reviews in Software Engineering" EBSE Technical Report, EBSE , [17] F. Scheuren, What is a Survey? December 21, 2004, [Online]. Available: [Accessed: September 2012]. 40

50 [18] J. Lee Kreader. Child care and early education research connections. August 2012, [Online]. Available: [Accessed: September 2012]. [19] B. Kitchenham, Procedures for performing systematic reviews, Keele, UK, Keele University, vol. 33, [20] G. Mateescu, W. Gentzsch, and C. J. Ribbens, Hybrid Computing-Where HPC meets grid and Cloud Computing, Future Generation Computer Systems, vol. 27, no. 5, p , May 2011 [21] Q. Huang, C. Yang, D. Nebert, K. Liu, and H. Wu, Cloud computing for geosciences: deployment of GEOSS clearinghouse on Amazon s EC2, in Proceedings of the ACM SIGSPATIAL International Workshop on High Performance and Distributed Geographic Information Systems, New York, NY, USA, 2010, pp [22] T. Gunarathne, T.-L. Wu, J. Qiu, and G. Fox, Cloud computing paradigms for pleasingly parallel biomedical applications, in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, New York, NY, USA, 2010, pp [23] Y. Zhai, M. Liu, J. Zhai, X. Ma, and W. Chen, Cloud versus in-house cluster: evaluating Amazon cluster compute instances for running MPI applications, in State of the Practice Reports, New York, NY, USA, 2011, pp [24] H. A. Duran-Limon, L. A. Silva-Banuelos, V. H. Tellez-Valdez, N. Parlavantzas, and M. Zhao, Using Lightweight Virtual Machines to Run High Performance Computing Applications: The Case of the Weather Research and Forecasting Model, in 2011 Fourth IEEE International Conference on Utility and Cloud Computing (UCC), Melbourne, Australia, 2011, pp [25] K. R. Jackson, L. Ramakrishnan, K. Muriki, S. Canon, S. Cholia, J. Shalf, H. J. Wasserman, and N. J. Wright, Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud, in 2010 IEEE Second International Conference on Cloud Computing Technology and Science (CloudCom), USA, 2010, pp [26] E. Deelman, G. Singh, M. Livny, B. Berriman, and J. Good, The cost of doing science on the cloud: The Montage example, in High Performance Computing, Networking, Storage and Analysis, Texas, 2008, pp [27] A. G. Carlyle, S. L. Harrell, and P. M. Smith, Cost-Effective HPC: The Community or the Cloud?, in 2010 IEEE Second International Conference on Cloud Computing Technology and Science (CloudCom), 2010, USA, pp [28] D. de Oliveira, K. Ocana, E. Ogasawara, J. Dias, F. Baiao, and M. Mattoso, A Performance Evaluation of X-Ray Crystallography Scientific Workflow Using SciCumulus, in 2011 IEEE International Conference on Cloud Computing (CLOUD),USA, 2011, pp [29] G. B. Berriman, G. Juve, E. Deelman, M. Regelson, and P. Plavchan, The Application of Cloud Computing to Astronomy: A Study of Cost and Performance, in 2010 Sixth IEEE International Conference on e-science Workshops, Brisbane, Australia 2010, pp [30] S. Ostermann, A. Iosup, N. Yigitbasi, R. Prodan, T. Fahringer, and D. Epema. A performance analysis of ec2 cloud computing services for scientific computing. In Cloud Computing: 1st Intl Conf. on Cloud Computing (CloudComp 2009), volume 34 of Lecture Notes (LNICST), pages Springer, [31] T. W. Wlodarczyk and Chunming Rong, An Initial Survey on Integration and Application of Cloud Computing to High Performance Computing, in 2011 IEEE Third International Conference on Cloud Computing Technology and Science (CloudCom), 2011,USA, pp [32] P. Mehrotra, J. Djomehri, S. Heistand, R. Hood, H. Jin, A. Lazanoff, S. Saini, and R. Biswas, Performance evaluation of Amazon EC2 for NASA HPC applications, in Proceedings of the 3rd workshop on Scientific Cloud Computing Date, New York, NY, USA, 2012, pp

51 [33] Amazon web services User Guide: Amazon EC2 Cost Comparison Calculator," March, [Online]. Available: [Accessed: September 2012]. [34] Evangelinos, C., C. Hill. "Cloud Computing for Parallel Scientific HPC Applications: Feasibility of Running Coupled Atmosphere-Ocean Climate Models on Amazon's EC2," The First Workshop on Cloud Computing and its Applications (CCA'08), October [35] M. R. Palankar, A. Iamnitchi, M. Ripeanu, and S. Garfinkel, Amazon S3 for science grids: a viable solution?, in Proceedings of the 2008 international workshop on Data-aware distributed computing, New York, NY, USA, 2008, pp [36] V. Subramanian, L. Wang, E.-J. Lee, and P. Chen, Rapid Processing of Synthetic Seismograms Using Windows Azure Cloud, in 2010 IEEE Second International Conference on Cloud Computing Technology and Science (CloudCom),USA, 2010, pp [37] K. Keahey, R. Figueiredo, J. Fortes, T. Freeman, M. Tsugawa Science Clouds: Early Experiences in Cloud Computing for Scientific Applications, in Proc. International conf of Chicago Conf. Circuit and System Theory, Chicago, Aug- 2008, pp [38] C. Vecchiola, S. Pandey, and R. Buyya, High-performance cloud computing: a view of scientific applications, in th International Symposium on Pervasive Systems, Algorithms, and Networks (ISPAN 2009), Dec. 2009, Piscataway, NJ, USA, 2009, pp [39] J. Li, M. Humphrey, D. Agarwal, K. Jackson, C. van Ingen, and Y. Ryu, escience in the cloud: A MODIS satellite data reprojection and reduction pipeline in the Windows Azure platform, in Parallel Distributed Processing (IPDPS), 2010 IEEE International Symposium on, USA, 2010, pp [40] P. Watson, P. Lord, F. Gibson, P. Periorellis, and G. Pitsilis, Cloud Computing for e- Science with CARMEN, in 2nd Iberian Grid Infrastructure Conference Proceedings, 2008, pp [41] C. Liu, X. Zhang, J. Chen, and C. Yang, An Authenticated Key Exchange Scheme for Efficient Security-Aware Scheduling of Scientific Applications in Cloud Computing, in 2011 IEEE Ninth International Conference on Dependable, Autonomic and Secure Computing (DASC), 2011,Sydny, Australia, pp [42] A. Pecchia, A. Sharma, Z. Kalbarczyk, D. Cotroneo, and R. K. Iyer, Identifying Compromised Users in Shared Computing Infrastructures: A Data-Driven Bayesian Network Approach, in Reliable Distributed Systems (SRDS), th IEEE Symposium on, 2011, pp [43] S. Ahmed and A. Abdullah, Telemedicine in a cloud ; A review, in Computers Informatics (ISCI), 2011 IEEE Symposium on, 2011, pp [44] J. C. Mace, A. Van Moorsel, P. Watson, and U. of N. upon T. C. Science, The case for dynamic security solutions in public cloud workflow deployments. Computing Science, Newcastle University, [45] M. Tsugawa, A. Matsunaga, and J. Fortes, User-Level Virtual Network Support for Sky Computing, in e-science, e-science 09. Fifth IEEE International Conference on, 2009, pp [46] S. Kamara and K. Lauter, Cryptographic Cloud Storage, in Financial Cryptography and Data Security, vol Springer Berlin Heidelberg, 2010, pp [47] L. Vaquero, L. Rodero-Merino, and D. Morn, "Locking the sky: a survey on IaaS cloud security," Computing, vol. 91, pp , [48] G. Cheng and A. K. Ohoussou, Sealed storage for trusted cloud computing, in 2010 International Conference on Computer Design and Applications (ICCDA), China, 2010, vol. 5, pp [49] N. Santos, K.P. Gummadi, and R. Rodrigues, Towards trusted cloud computing. In Proceedings of the Workshop on Hot Topics in Cloud Computing, San Diego, CA, USA, June

52 [50] M. Aslam, C. Gehrmann, L. Rasmusson, and M. Bj orkman, Securely launching virtual machines on trustworthy platforms in a public cloud, in CLOSER 2012 Proceedings of the 2nd International Conference on Cloud Computing and Services Science, Porto, Portugal, April, [51] J. Schiffman, T. Moyer, H. Vijayakumar, T. Jaeger, and P. McDaniel, Seeding clouds with trust anchors, in Proceedings of the 2010 ACM workshop on Cloud computing security workshop, New York, NY, USA, 2010, pp [52] R. Tudoran, A. Costan, G. Antoniu, and L. Bougé, A performance evaluation of Azure and Nimbus clouds for scientific applications, in Proceedings of the 2nd International Workshop on Cloud Computing Platforms, New York, NY, USA, 2012, pp.1 6. [53] A. Iosup, S. Ostermann, M. N. Yigitbasi, R. Prodan, T. Fahringer, and D. H. J. Epema, Performance Analysis of Cloud Computing Services for Many-Tasks Scientific Computing, IEEE Transactions on Parallel and Distributed Systems, vol. 22, no. 6, pp , Jun [54] E. Walker, "Benchmarking amazon ec2 for high-performance scientific computing," in USENIX ;login: magzine, Oct [55] Y. Shi, X. Meng, J. Zhao, X. Hu, B. Liu, and H. Wang, Benchmarking cloud-based data management systems, in Proceedings of the second international workshop on Cloud data management, New York, NY, USA, 2010, pp [56] J. Napper and P. Bientinesiy, Can cloud computing reach the TOP500?, in Combined Workshops on UnConventional High Performance Computing Workshop Plus Memory Access Workshop, UCHPC-MAW 09, Co-located with the 2009 ACM International Conference on Computing Frontiers, CF 09, 18-20, May 2009, Ischia, Italy, 2009, pp [57] Q. He, S. Zhou, B. Kobler, D. Duffy, and T. McGlynn, Case study for running HPC applications in public clouds, in 19th ACM International Symposium on High Performance Distributed Computing, HPDC 2010, June 21, June 25, 2010, Chicago, IL, United states, 2010, pp [58] C. H. Constantinos Evangelinos, "Cloud Computing for parallel Scientific HPC Applications: Feasibility of Running Coupled Atmosphere-Ocean Climate Models on Amazon's EC2," The First Workshop on Cloud Computing and its Applications (CCA'08), October [59] S. Akioka and Y. Muraoka, HPC benchmarks on Amazon EC2, in 24th IEEE International Conference on Advanced Information Networking and Applications Workshops, WAINA 2010, April 20, April 23, 2010, Perth, Australia, 2010, pp [60] R. Iakymchuk, J. Napper, and P. Bientinesi, Improving high-performance computations on clouds through resource underutilization, in Proceedings of the 2011 ACM Symposium on Applied Computing, New York, NY, USA, 2011, pp [61] D. Ghoshal, R. S. Canon, and L. Ramakrishnan, I/O performance of virtualized cloud environments, in Proceedings of the second international workshop on Data intensive computing in the clouds, New York, NY, USA, 2011, pp [62] H. Chen, S. Wu, X. Shi, H. Jin, and Y. Fu, LCM: A lightweight communication mechanism in HPC cloud, in th International Conference on Pervasive Computing and Applications (ICPCA), Port Elizabeth, South Africa,2011, pp [63] P. Brereton, B. A. Kitchenham, D. Budgen, M. Turner, and M. Khalil, Lessons from applying the systematic literature review process within the software engineering domain, Journal of Systems and Software, vol. 80, no. 4, pp , Apr [64] D. Jayasinghe, S. Malkowski, Q. Wang, J. Li, P. Xiong, and C. Pu, Variations in Performance and Scalability When Migrating n-tier Applications to Different Clouds, in 2011 IEEE International Conference on Cloud Computing (CLOUD), Washington DC, USA, 2011, pp [65] C. Wohlin, P. Runeson, M. Host, C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in Software Engineering: an Introduction. Kluver Academic Publishers,

53 [66] William M.K. Trochim., Research Based Knowledge Base Internal validity. October, 2006, [Online]. Available: [Accessed: September 2012]. [67] William M.K. Trochim. Research Based Knowledge Base External validity. October, 2006 [Online]. Available: September 2012]. [68] William M.K. Trochim. Research Based Knowledge Base Conclusion validity. [Online]. Available: [Accessed: September 2012]. [69] Muhammad Umar Farooq. Usman Farooq, "Exploring the Benefits and Challenges of Applying Agile Methods in Offshore Development," Blekinge Institute of Technology, Karlskrona, Sweden., Master Thesis MSE-2010:39, December [70] N. Mack, C. Woodsong, K.M. Macqeen, G. Guest and E. Namey, Qualitative research methods: A data collector s field guide. North Carolina, Family Health International, [71] Amazon Web services, How AWS Price Works, December 2011, [Online]. Available: September 2012]. 44

54 APPENDIX A Some of questionnaires: In which cases do you expect that the cost for running an HPC application in the cloud would be lower than buying a super computer and run the application on that? In which cases do you expect that the cost for running an HPC application in the cloud would be higher than buying a super computer and run the application on that? Can public Cloud be used for High Performance Computing? What are the different kinds of security solutions used by different Cloud vendors for HPC? What are the different kinds of tools that are used to measure the performance of HPC in Cloud? 45

55 Search Strings: APPENDIX B RQ1: What are the cost and performance aspects of HPC applications in the Cloud? (HPC applications OR high performance computing applications OR scientific applications) AND (cloud computing OR on demand computing) AND (cost OR monetary OR price OR economics) AND (performance OR quality) RQ2: Which HPC applications have been executed in the Cloud? (cloud computing OR on demand computing)and (high performance computing applications OR supercomputer applications OR scientific applications OR HPC applications) AND (executed OR solved OR shifted OR moved OR processing) RQ3: What type of security issues are involved in HPC in Cloud? ( HPC OR hpc applications OR high performance computing applications OR scientific applications) AND ( cloud computing OR on demand computing ) AND (Security OR issues OR problems OR Threats) RQ4: What are the different kinds of tools that are used to measure the performance of HPC in Cloud? ( HPC OR hpc applications OR high performance computing applications OR scientific applications) AND (cloud computing OR on demand computing) AND (tools OR benchmark) AND (performance) 46

56 APPENDIX C 47

57 48

58 49

59 50