A Review of Service Selection in Cloud Computing

Transcription

1 4 th World Conference on Applied Sciences, Engineering & Technology October 2015, Kumamoto University, Japan A Review of Service Selection in Cloud Computing MUHAMMAD-BELLO BILKISU LARAI 1, 2 1 Department of Computer Science and Electrical Engineering, Kumamoto University, Japan 2 Department of Information & Media Technology, Federal University of Technology Minna, Nigeria blabdool@dbms.cs.kumamoto-u.ac.jp, bilkisu_bello@futminna.edu.ng Abstract: Cloud computing has enabled the on-demand utilization of elastic and scalable Information Technology (IT) infrastructure from remote centers worldwide. With its emergence, organizations no longer need to make huge upfront investment for the procurement of IT infrastructure. The adoption of cloud services is rising rapidly and this has engendered many Cloud Service Providers (CSPs) into the market. This trend has made it more difficult for prospective cloud users to select an optimum service from a pool of functionally equivalent CSPs. Consequently, this predicament has actuated research in cloud service selection where different techniques have been employed in ranking of CSPs based on different number of criteria. A systematic mapping study is presented in this paper in order to present a wide review of primary studies in the area of service selection in cloud computing. The initial search identified 356 studies, of which 33 were identified as relevant to answer the research questions after undergoing the review quality assessment and applying the inclusion/exclusion criteria. The study was based on 4 main research questions in order to establish the research evidences existing on the topic and also provide an indication of the quantity of evidence. The findings show that most of the primary studies have focused on Infrastructure as a Service (IaaS). Also, different frameworks have been developed using different sets of evaluation criteria based on different mathematical techniques with Analytic Hierarchy Process (AHP) being the most common. The results also show that there is currently no unified or defined metrics/evaluation criteria for comparing cloud service providers. In conclusion, the research findings show that current research in the area of service selection in cloud computing have concentrated on mainly the quantitative criteria in their evaluation and selection process while the qualitative criteria of Quality of Service (QoS) are yet to be adequately considered. This paper also contributes to summarizing the body of knowledge in the area of service selection in cloud computing and directs effort for future research in the area. Keywords: Cloud Services, Service Selection in Cloud computing, Cloud Computing, Mapping study, Cloud Service Selection Introduction: Over recent years, a common understanding of cloud computing has continued to evolve. Cloud computing involves the utilization of Information Technology (IT) infrastructures that possess scalable and elastic IT-enabled capabilities to deliver services to external customers [1]. These IT-enabled capabilities entail the ubiquitous, convenient on-demand network access to a shared pool of configurable computer resources such as; servers, storage, networks, applications, and services which can be speedily provisioned and deployed with minimum management effort or service provider interaction [2]. With its emergence, Cloud Service Providers (CSPs) are able to provide reliable infrastructure to host instances of clients operating systems, as well as hosted solutions as services over the internet to cloud users from remote datacenters worldwide. There are three main service models primarily used in the deployment of cloud services by CSPs [3][4]. The first service model is Infrastructure as a Service (IaaS) which provides an environment for cloud users to deploy, run and manage virtual computing resources such as virtual machines, storage and servers. Examples of IaaS include Rackspace, Google Compute Engine, Amazon Elastic Cloud Compute (EC2) and Microsoft Azure (Virtual Machines). IaaS also provides on demand incremental scalability of computing resources and storage enabling cloud users to scale up or down as required [5]. The second service model is Platform as a Service (PaaS) which basically provides a platform for users to develop and deploy their own software and applications. Examples of PaaS include Google App Engine (GAE), and Microsoft Azure. The third main service model is Software as a Service (SaaS) which provides cloud users with access to cloud hosted software applications via the internet. Examples of SaaS include Google apps, Microsoft Office 365 and Amazon Relational Database Service (RDS). Many organizations make excess provision for their IT infrastructure by at least 100% of what is ideally required. This excess provisioning was to provide necessary IT infrastructure for peak utilization periods and, also to add safety margins to the existing provision [4][6]. Thus, additional cost is often incurred by Small and Medium Enterprises (SMEs) in order to make provision for these IT infrastructures. The emergence of cloud computing has revolutionized the IT industry and also impacted SMEs worldwide. This has promoted cloud WCSET Copyright 2015 BASHA RESEARCH CENTRE. All rights reserved

2 MUHAMMAD-BELLO BILKISU LARAI computing as an emerging trend and a possible cost saving solution in comparison to the capital intensive over-provisioning of IT infrastructures by many organizations. However, the popularity of cloud computing and cloud services has engendered many CSPs that offer different cloud services such as SaaS, PaaS and IaaS to users. It has also been argued that the barrier to entry for CSPs is low [6]. This means there are more choices available in the market place for prospective cloud users. However, this trend poses a challenge for the cloud users to select an optimum service from an abundance of possibilities. There are other challenges that prevent cloud users from leveraging on the full potentials of cloud computing. For example, applications often have specific functional requirements which need to be satisfied by a prospective CSP before it can be deployed on their cloud infrastructure. It has also been recently argued that the number of cloud service providers with similar functionalities has continued to increase [7]. This has also made it difficult for cloud service users to select the most suitable cloud service from a pool of functionally equivalent CSPs. It is worthy to note that these cloud services vary substantially in terms of some of the quantifiable criteria of Quality of Service (QoS) such as cost and performance. Consequently, this predicament has actuated research in the area of cloud service selection where different mathematical techniques have been employed in ranking of CSPs based on different criteria. This paper presents a systematic mapping study which is a broad review of relevant primary studies related to service selection in cloud computing. Its significance is to provide a summary of research efforts in this area with emphasis on identifying gaps in knowledge and open areas for future research. It aims to identify the evidences available on the current state of research related to service selection in cloud computing. The objectives are as follows: 1. To identify the techniques employed in past studies and acquire evidence as to which techniques have been the most employed in cloud service selection and their limitations. 2. To identify, assess and classify the evaluation criteria that are being employed in cloud service selection. 3. To identify the existing frameworks for evaluating and comparing services of CSPs. The rest of this paper is organized as follows: The next section describes the research questions and review methodology. This is followed by the presentation of results for the study and discussions. The final section concludes the paper. Materials and Methods: This systematic mapping study employed the guidelines given for software engineering literature review by Kitchenham & Charter [8]. Following these guidelines, four research questions (RQs) were specified to guide this study. A. Research Questions 1. RQ1: What evaluation criteria have been employed in evaluating and selecting cloud services in the past researches? RQ1 aims to identify and categorize the criteria that have been employed in the research effort so far. This will assess the types of criteria that are being used and categorize them appropriately. 2. RQ2: Which cloud service models are considered by the past studies? RQ2 aims to highlight the scope of the available studies in service selection in cloud computing regarding which service model has been targeted. This information will highlight the areas that have received considerable amount of research effort and also identify those requiring additional effort. 3. RQ3: What techniques have been employed in the evaluation and selection of cloud services? RQ3 aims to identify and categorize the techniques that have been employed in the evaluation and ranking of cloud service providers. 4. RQ4: What frameworks/models have been developed in the past studies? This aims to identify any existing framework/models built. B. Search Strategy The following search strategy was employed in this study to discover as many studies related to the research topic and research questions. 1. Literature Resources Cloud computing being an emerging technology, research focus in the area of service selection in cloud computing has increased over the past few years. Consequently, relevant materials for the study are scattered across different journals. The following digital libraries were searched in order to provide a comprehensive bibliography for the primary review: o Elsevier Science Direct o IEEEXplore Digital Library o ACM Digital Library o Springer Link and Springer Conference Proceedings In addition, the reference lists of relevant studies identified were used as pointers to other relevant studies. 2. Search String The advanced search feature provided by the digital libraries was used to construct the search string. Major distinct terms from the research questions as well as acronyms and their full meanings were used as keywords. The search string was updated with keywords identified from relevant studies. Using this

3 A Review of Service Selection in Cloud Computing strategy, the search string depicted in Table1 was attained. TABLE1. THE SEARCH STRING Study Identifier Table 2. The Data Extraction Form ( Cloud computing* OR Cloud Selection,OR QoS ranking OR Optimal Cloud Service OR Cloud Service Selection OR Evaluation criteria AND IaaS OR SaaS OR PaaS OR Framework* OR Approach,) The OR and AND operators were used with these search terms to maximize the amount and relevance of search results. Subsequently, the following Inclusion/Exclusion criteria were used to select the final papers for review. 3. Inclusion/Exclusion Criteria o Only those articles related to service selection/ranking of cloud services were selected, as these were the most appropriate and the focus of this review. o Only those articles which clearly described the technique used in selecting/ranking cloud o services were selected. Only the most recent or complete study was selected from duplicate publications of the same study. o Unpublished working papers, masters and doctoral dissertations were excluded since journal and conference papers are the most appropriate channels for disseminating and acquiring new findings in the academia. 4. Data Extraction and Quality Assessment The data extraction form depicted in Table 2 was used to gather all the necessary information required in order to answer the research questions. In addition to the data extraction process, a quality assessment form was also used to assess the study quality of the selected primary studies. Four (4) quality assessment criteria were defined as shown in Table 3. The study quality assessment was necessary because it would aid in the selection of the most relevant primary studies and also the data analysis and synthesis process. Table 3. Quality Assessment Criteria QA1: Are the study aims clearly stated? QA2: Are the variables/evaluation (QoS) criteria used in the study adequately measured? QA3: Are the measures used in the study clearly defined? QA4: Are the study contributions clearly stated? Title Authors Year of Publication Literature Source Literature Type RQ1: Evaluation Criteria used RQ2: Cloud Service Models targeted RQ3: Technique adopted RQ4: Framework developed Additional Notes 5. Data Synthesis The data extracted were collated and summarized using tables and charts in a manner that is consistent with the research questions. Furthermore, descriptive synthesis was used to discuss the results and elaborate the findings. Results and Discussion: The initial search resulted in 356 candidate primary studies identified from the aforementioned literature sources. After applying the inclusion/exclusion criteria, 56 candidate primary studies were identified but only 42 were available for download. The quality assessment criteria further pruned down the number to 33 which were eventually selected as relevant primary studies. This is because primary studies with a low quality assessment (less than 50%) were excluded from the final list. Figure 1 depicts the selection process and presents the number of studies identified at each step. The quality assessment criteria were scored as follows: YES =1, PARTIALLY = 0.5, NO = 0. Each criterion could have only one score depending on the answer to the question. The scores for each study were aggregated to determine the quality score for that study. These scores were adjusted as a percentage. Table 4 summarises the quality scores for the selected primary studies. Figure 2 shows the distribution of the selected primary studies per year. Set I STEP1 Set II STEP2 Set III Literature Search Results Figure 1: Selection Process to identify relevant studies

4 MUHAMMAD-BELLO BILKISU LARAI Table 4.Summary of Quality Assessment Scores Quality Score Number of Studies Study ID 100% 5 [S1],[S5],[S12],[S13],[S14] 87.50% 1 [S27] 75% 13 [S4],[S7],[S8],[S16],[S17], [S18],[S19],[S20],[S25],[S26], [S28],[S30],[S33] 62.50% 9 [S2],[S3],[S6],[S9],[S10],[S21], [S29],[S31],[S32] 50% 5 [S11],[S15],[S22],[S23],[S24] Total 33 Table 5. Categorization of Evaluation Criteria and Scores Category Number of Percentage studies Score Quantitative 26 79% Qualitative 0 0% Quantitative & 7 21% Qualitative As there is no unified metric defined for the key performance indicators used as evaluation criteria in cloud service selection, different studies used different criteria. While some studies [9][10]made an effort to define the meaning of the criteria used, majority did not which makes it difficult to determine if the defined variables are valid and reliable in the context in which they have been used. There were however some common criteria that were considered in most of the primary studies. Table 6 presents a summary of these evaluation criteria that have been employed in most of the papers. Table 6.Evaluation and Selection Criteria Figure 2: Distribution of selected primary studies per year RQ1: What evaluation criteria have been employed in evaluating and selecting cloud services in the past researches? RQ1 aims to identify and categorize the evaluation and selection criteria that have been employed in the research effort so far. The result shows that there are currently no unified or defined metrics/evaluation criteria for comparing cloud service providers. Consequently different studies used different sets of criteria in evaluating and ranking CSPs. Majority of the studies employed only qualitative criteria such as performance and cost. This can be attributed to the fact that qualitative criteria are objective in nature and can be measured using hardware and software monitoring tools. On the other hand, no single study adopted only the qualitative criteria. Qualitative criteria such as user experience, Vendor reputations etc. are subjective in nature. Therefore, measuring such criteria is not a straight forward task. However, a combination of both quantitative and qualitative criteria was considered in few recent studies. A summary of studies showing the categorization of the evaluation criteria is shown in Table 5. Evaluation Criteria No of occurrence in primary studies Cost 21 28% Performance 12 16% Availability 9 12% Reliability 8 11% Security 7 9% Usability 6 8% Agility 6 8% Reputation 5 7% Cost is the most employed evaluation criteria in the selected primary studies followed by Performance. This shows that majority of the existing research literature in cloud service selection focused mostly on comparing CSPs based on cost or performance benchmarks. It is important to note that some of these performance metrics used in evaluating cloud services have other sub criteria which are also considered. However, the details are beyond the scope of this study. They have been described in the work of Costa et al. [10] and Garg et al. [9]. In addition, missing criteria, redundant and less objective criteria were observed in some current approaches. These flaws can be improved upon in future work. RQ2: Which cloud service models are considered by the past studies? RQ2 aims to highlight the scope of the available studies regarding which cloud service model has been targeted in the past studies. The result shows %

5 A Review of Service Selection in Cloud Computing that while majority of the studies addressed cloud service selection from the perspective of a single type of service with IaaS being the most common, others adopted a hybrid approach and targeted all the service types. The result as depicted in Figure 3 shows that 40% of the studies focused on IaaS, 27% focused on SaaS while none of the studies reviewed targeted PaaS. The result shows that 21% of the studies were unspecific. This was attributed to the fact that these studies were based on assumptions that the models developed can be easily adopted across any types of cloud service model. The major shortcoming of the studies that developed such models is the fact that they failed to recognize the fact that certain evaluation criteria are only peculiar to a particular service model. Therefore, adopting such models across all platforms might not be reliable or valid in some cases. Figure 3: Distribution of the cloud service models targeted RQ3: What techniques have been employed in the evaluation and selection of cloud services? RQ3 aims to identify and categorize the techniques that have been employed in the evaluation and ranking of cloud service providers. The results show that Analytic Hierarchy Process (AHP) is the most common technique adopted as 25% of the studies used this technique. This is followed by Fuzzy based Techniques such as Fuzzy AHP, Fuzzy TOPSIS and Fuzzy Simple Additive Weighing (SAW) which were adopted in 17% of the studies. In general, it was observed that Multicriteria Decision Analysis (MCDA) techniques were the most employed in past studies. This is because AHP, Fuzzy, TOPSIS, SAW, Optimization, Outranking, SAW and Analytic Network Process (ANP) are all variants of MCDA. The result also shows that new techniques have been adopted in other studies. Techniques such as Integer Programming, Primitive Cognitive Network Process (P-CNP), Probability Theory, Satisfiability Modulo Theories (SMT), Singular Value Decomposition Technique (SVD), Weighted Difference and Exponential Weighted Difference, Case-Based Reasoning, Forward Checking Based Backtracking and Genetic-Based technique have also been utilized. However, it was observed that these techniques do not seem to be popular in this domain as they have only been adopted in a single study as depicted in Table 7. Therefore, it is imperative to study these techniques in more details in order to ascertain their validity and reliability in this research domain. Table7. Evaluation techniques adopted in Cloud Service Selection Evaluation Study ID No of % Techniques Studies AHP Based Fuzzy Based [S1],[S5],[S8],[S17], [S21],[S24],[S25], [S26],[S32] [S2],[S14],[[S21], [S24],[S28],[S32] 9 25% 6 17% TOPSIS [S17],[S18],[S21], 4 11% Based [S28] Optimization [S12],[S19],[S32] 3 8% Based SAW Based [S14],[S29] 2 6% MCDM [S17],[S30] 2 6% ANP Based [S11] 1 3% Outranking [S3] 1 3% Integer Programming Primitive Cognitive Network Process (P- CNP) Probability Theory Satisfiability Modulo Theories (SMT) Singular Value Decompositio n Technique (SVD) Weighted Difference and exponential weighted difference Case-Based Reasoning Forward Checking, Backtracking and Genetic- Based [S33] 1 3% [S31] 1 3% [S2] 1 3% [S6] 1 3% [S23] 1 3% [S16] 1 3% [S20] 1 3% [S27] 1 3% RQ4: What frameworks/models have been developed in past studies? This research question aims to identify any existing framework/models developed in past studies. The results revealed that only10 papers representing 30% of past studies have attempted to develop a framework/model. The remaining 23 papers representing 70% of the primary studies did not

6 MUHAMMAD-BELLO BILKISU LARAI develop a framework. Figure 4 shows the distribution of studies that developed frameworks as against those that did not. A description of the frameworks developed in the 10 studies identified is summarized in Table 8. Table 8. Frameworks Developed In Past Studies Description of framework/models Developed a semantic-based SaaS discovery and selection system and a unified ontology for integrating service domain knowledge, SaaS cloud characteristics, QoS metrics and real SaaS offers. SID [S4] SMICloud: Systematically measures all the QoS attributes proposed by Cloud Service Measurement Index Consortium (CSMIC) and rank CSPs based on the attributes Proposed an improvement to a previous model (Skyline algorithm) which allows searching and selecting CSPs based on user's requirements. [S1] [S3] Figure 4: Distribution of studies that developed frameworks/model Conclusion: An overview of past research studies in the area of cloud service selection has been presented in this study. This is the first identifiable systematic mapping study in the area of service selection in cloud computing. In this paper, quality assessment criteria were applied in the selection of the primary studies. Available evidences in past literatures as to the current state of affairs in this research area have been presented in this study by answering four main research questions which will guide future research. This study has been able to identify and classify the techniques employed in service selection in cloud computing. The evaluation criteria have been identified, assessed and classified. In this study, the existing models/frameworks developed have also been identified. The findings show that most of the primary studies have focused on IaaS, and different frameworks have been developed based on different mathematical techniques with AHP being the most common. The focus of majority of the past studies has been on objective cost performance analysis and benchmark testing while the subjective assessment of the actual cloud users haven t been given adequate attention. The results also show that currently, there are no unified or defined metrics/evaluation criteria for comparing cloud service providers. In conclusion, current research in the area of service selection in cloud computing has concentrated on mainly the quantitative criteria in their evaluation and selection process while the qualitative criteria are yet to be adequately considered. Future work in this area will be focused on developing a framework that adequately captures the subjective qualitative evaluation criteria as well as the objective evaluation criteria in a meaningful way by defining appropriate metrics that best describes the service models targeted. QuRAMRecommender: Maintains a case base of previous CSP selections to select a cloud service that best matches a customer s preferences and requirements. Proposed a framework for enterprises to recommend and select CSPs automatically based on their business requirements and CSP offerings Proposed a cloud performance monitoring system that captures and utilizes user feedbacks in cloud service selection MADMAC: Framework consists of Cloud Switch, Cloud Type and Vendor Choice as decision areas for cloud adoption. Proposed a fuzzy MCDM framework that identifies required services and determines criteria weights for Cloud service selection CCCloud: a context-aware and credible cloud service selection model based on the objective assessments from quantitative performance testing parties and the comparison and aggregation of subjective assessments extracted from ordinary cloud consumers. Model aggregates information from user feedbacks and objective performance analysis from trusted third party for cloud service selection. [S20] [S30] [S22] [S29] [S21] [S13] [S14] References: [1] Kumar A., (2010), Creating Dynamic Infrastructure: Cloud Computing and Integrated Managed Services for Smarter Organization, Insights from IBM featuring Gartner Research, Vol.1, No. 4, November [2] Mell P., Grance T., (2011) The NIST Definition of Cloud Computing, National Institute of Standards and Technology Special. Publication , 2011, pp1-7 [3] Jula A., Sundararajan E., Othman Z., Cloud computing service composition: A systematic literature review, Expert Systems with Applications, Vol. 41, No. 8, pp , [4] G. Eugene, Cloud Computing Models, Working Paper CISL# , January, 2013 [Online] Available:

7 A Review of Service Selection in Cloud Computing [5] Buyya R., Yeo C.S., Venugopal S., Broberg J., Brandic I, (2009) Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility, Future Generation Computer Systems, Vol. 25, No. 6, p. 17, June 2009, pp [6] Kepes B., Moving your Infrastructure to the Cloud- How to Maximize Benefits and Avoid Pitfalls, Rackspace Support, [Online]. Available: tes/default/files/whitepaper_pdf/movingyourinf rastructuretothecloud- HowtoMaximizeBenefitsandAvoidPitfalls.pdf. [Accessed: 19-Sep-2015]. [7] Alabool H.M. Mahmood K., (2013), Review on cloud service evaluation and selection methods, International Conference on Research and Innovation in Information Systems, ICRIIS, ISBN August 2015, pp [8] Kitchenham B. Charters S., (2007) Guidelines for performing Systematic Literature Reviews in Software Engineering, Engineering EBSE Technical Report Version 2.3. [9] Garg S. K.,Versteeg S., Buyya R., (2013) A framework for ranking of cloud computing services Future Generation Computer Systems. ISSN X, Vol.29 No. 4, pp [10] Costa P., Santos J. P., Silva M. M., (2013) Evaluation criteria for cloud services IEEE International Conference on Cloud Appendix A: List of Selected Primary Studies 1. Garg S. K.,Versteeg S., Buyya R., (2013) A framework for ranking of cloud computing services Future Generation Computer Systems. ISSN X, Vol.29 No. 4, pp Olga G., Dessislava., P.A., (2014) QoS-Aware Web Service Selection Accounting for Uncertain Constraints th EUROMICRO Conference on Software Engineering and Advanced Applications, ISBN , No. 2014, pp Abourezq M., Idrissi A., (2014) Introduction of an outranking method in the Cloud Computing Research and Selection System based on the Skyline 2014 IEEE Eighth International Conference on Research Challenges in Information Science (RCIS), ISBN , pp Afify Y. M., Moawad I. F., Badr N. L., Tolba M. F., (2013) A semantic-based Software-as-a- Service (SaaS) discovery and selection system Proceedings th International Conference on Computer Engineering and Systems, ICCES 2013, ISBN , pp Karim R., Ding C. Miri A., (2013) An End-to- End QoS Mapping Approach for Cloud Service Selection Services (SERVICES), 2013 IEEE Ninth World Congress on Services, ISBN , pp Chen C.,Yan S., Zhao G., Lee B.S., Singhal S., (2012) A systematic framework enabling automatic conflict detection and explanation in cloud service selection for enterprises Proceedings IEEE 5th International Conference on Cloud Computing, CLOUD 2012, ISSN , pp Costa P., Santos J. P., Silva M. M., (2013) Evaluation criteria for cloud services IEEE International Conference on Cloud Computing, CLOUD, ISSN , pp Godse M., Mulik S., (2009) An approach for selecting Software-as-a-Service (SaaS) product CLOUD IEEE International Conference on Cloud Computing, ISBN , pp Gupta S., Muntes-Mulero V., Matthews P., Dominiak J. Aida O., Aranda J., Seycek S., (2015) Risk-Driven Framework for Decision Support in Cloud Service Selection th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, ISBN , pp Karim R., Ding C., Miri A., Liu X., (2014) End-to-end QoS Mapping and Aggregation for Selecting Cloud Services Proceedings of CTS'14 (2014), ISBN , pp Le S., Dong H., Hussain F.K., Hussain O.K., Ma J., Zhang Y., (2014) Multicriteria Decision Making with Fuzziness and Criteria Interdependence in Cloud Service Selection 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), ISBN , pp Qian H., Zu H., Cao C., Wang Q. (2013) CSS: Facilitate the cloud service selection in IaaS platforms Proceedings of the 2013 International Conference on Collaboration Technologies and Systems, CTS 2013, ISBN , pp Qu L., Wang Y., Orgun M.A., Liu L., Liu H., Bouguettaya A., (2015) CCCloud : Context- Aware and Credible Cloud Service Selection Based on Subjective Assessment and Objective Assessment IEEE Transactions on Services Computing, May/June 2015, Vol 8. No. 3, pp Qu L., Wang Y., Orgun, M.A., (2013) Cloud Service Selection Based on the Aggregation of User Feedback and Quantitative Performance Assessment 2013 IEEE International

8 MUHAMMAD-BELLO BILKISU LARAI Conference on Services Computing, ISBN , pp Raj R.J.R., Sasipraba T., (2010) Web service selection based on QoS Constraints Trendz in Information Sciences & Computing (TISC), 2010 Vol. 6, pp Rehman Z.U. Hussain F.K., Hussain O.K., (2011) Towards Multi-criteria Cloud Service Selection 2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing ISBN , pp Rehman Z.U. Hussain F.K., Hussain O.K., (2012) Iaas cloud selection using MCDM methods Proceedings - 9th IEEE International Conference on E-Business Engineering, ICEBE 2012, ISBN , pp Rehman Z.U. Hussain F.K., Hussain O.K., (2013) Multi-Criteria IaaS service selection based on QoS history Proceedings - International Conference on Advanced Information Networking and Applications, AINA, ISSN X, pp Salama M., Shawish A., Zeid A., Kouta M., (2012) Integrated QoS utility-based model for cloud computing service provider selection Proceedings - International Computer Software and Applications Conference, ISSN , pp Soltani S., Martin P., Elgazzar K., (2014) QuARAMRecommender: Case-Based Reasoning for IaaS Service Selection The International Conference on Cloud and Autonomic Computing (CAC 2014), ISBN , pp Sun L., Dong H., Hussain F.K., Hussain O.K., Ma J., Zhang Y., (2014) A Hybrid Fuzzy Framework for Cloud Service Selection 2014 IEEE International Conference on Web Services, ISBN , pp Rehman Z.U., Hussain O.K., Parvin S., Hussain F.K., (2012) A Framework for User Feedback Based Cloud Service Monitoring 2012 Sixth International Conference on Complex, Intelligent, and Software Intensive Systems, ISBN , pp Chan H., Chieu T., (2010) Ranking and mapping of applications to cloud computing services by SVD 2010 IEEE/IFIP Network Operations and Management Symposium Workshops, ISSN , pp Chen C.T., Lin K.H., (2011) A decisionmaking method based on interval-valued fuzzy sets for cloud service evaluation 4th International Conference on New Trends in Information Science and Service Science (NISS), 2010, ISBN , pp Chen., Bai X., Huang X., Li M., Zhou L., (2011) Evaluating services on the cloud using ontology QoS model Proceedings - 6th IEEE International Symposium on Service-Oriented System Engineering, SOSE 2011, ISBN , pp Chen Y., Zhu Y., (2011) SaaS vendor selection basing on analytic hierarchy process Proceedings - 4th International Joint Conference on Computational Sciences and Optimization, CSO 2011, ISBN , pp Dastjerdi A.V., Garg S.K., Buyya R., (2011) QoS-aware deployment of network of virtual appliances across multiple Clouds Proceedings rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011, ISBN , pp Lo C.C., Chen, D.Y., Tsai C.F., Chao K.M., (2010) Service Selection Based on Fuzzy TOPSIS Method 2010 IEEE 24th International Conference on Advanced Information Networking and Applications Workshops (WAINA), ISBN , pp Saripalli P., Pingali G., (2011) MADMAC: Multiple attribute Decision methodology for Adoption of clouds Proceedings IEEE 4th International Conference on Cloud Computing, CLOUD 2011, ISSN , pp Yan S., Chen C., Zhao G. Lee B., (2012) Cloud service recommendation and selection for enterprises 8th international conference on Network and service management (cnsm) and 2012 workshop on systems virtualiztion management (svm), ISBN , pp Yuen K.K.F., (2012) Software-as-a-Service evaluation in cloud paradigm: Primitive cognitive network process approach 2012 IEEE International Conference on Signal Processing, Communications and Computing, ICSPCC 2012, ISBN , pp Zhang M., Ranjan R., Haller A., Georgakopoulos D. Strazdins P., (2013) Investigating decision support techniques for automating Cloud service selection CloudCom Proceedings: th IEEE International Conference on Cloud Computing Technology and Science, ISBN , pp Wang S., Zheng Z., Sun Q., Zou H., Yang, F., (2011) Cloud model for service selection 2011 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), ISBN , August 2015, pp