The Journal of Systems and Software

Transcription

1 The Journal of Systems and Software 84 (2011) Contents lists available at ScienceDirect The Journal of Systems and Software journal homepage: Framework for evaluation and selection of the software packages: A hybrid knowledge based system approach Anil S. Jadhav a,, Rajendra M. Sonar b a Computer Department, Sinhgad Institute of Management, Vadgaon (Bk.), Pune , India b Shailesh J. Mehta School of Management, Indian Institute of Technology Bombay, Mumbai , India article info abstract Article history: Received 19 November 2010 Received in revised form 10 February 2011 Accepted 11 March 2011 Available online 21 March 2011 Keywords: Software evaluation Software selection Knowledge based system Evaluation and selection of the software packages is complicated and time consuming decision making process. Selection of inappropriate software package can turn out to be costly and adversely affects business processes and functioning of the organization. In this paper we describe (i) generic methodology for software selection, (ii) software evaluation criteria, and (iii) hybrid knowledge based system (HKBS) approach to assist decision makers in evaluation and selection of the software packages. The proposed HKBS approach employs an integrated rule based and case based reasoning techniques. Rule based reasoning is used to capture user needs of the software package and formulate a problem case. Case based reasoning is used to retrieve and compare candidate software packages with the user needs of the package. This paper also evaluates and compares HKBS approach with the widely used existing software evaluation techniques such as analytic hierarchy process (AHP) and weighted scoring method (WSM) Elsevier Inc. All rights reserved. 1. Introduction The demand for reliable and qualitative software packages is continuously growing. In response to meet the growing demand software firms have been producing variety of software packages that are customizable and tailored to meet specific needs of the organization. Selection of inappropriate software package adversely affects business processes and functioning of the organization. The task of software package selection has become more complex due to (i) difficulty in accessing applicability of software packages to the business needs of the organization due to availability of large number of software packages in the market, (ii) existence of incompatibilities between various hardware and software systems, (iii) lack of technical knowledge and experience to decision makers, and (iv) ongoing improvements in information technology (Lin et al., 2006). The task of software selection is often assigned under schedule pressure and evaluators may not have time or experience to plan selection process in detail. Therefore, they may not use most appropriate method for software selection (Kontio et al., 1996). Thus, selecting a software package that meets specific needs of the organization is complicated and time consuming decision making process. This has led researchers to investigate better ways of evaluating and selecting software packages. Corresponding author. Tel.: ; fax: addresses: a s [email protected] (A.S. Jadhav), rm [email protected] (R.M. Sonar). Multi criteria decision making (MCDM) problem refers to making preference decisions over the available alternatives that are characterized by multiple, usually conflicting, attributes (Yoon and Hwang, 1995). Goal of the MCDM is: (i) to help decision makers to choose the best alternative, (ii) to sort out the alternatives that seems among the set of available alternatives, and (iii) to rank the alternatives in decreasing order of their performance (Mollaghasemi and Pet-Edwards, 1997). The process of evaluation and selection of the software packages involves simultaneous consideration multiple attributes to rank the available alternatives and select the best one. Therefore, evaluation and selection of the software packages can be considered as multi criteria decision making problem. Knowledge based systems (KBS) are computer based information systems that represents knowledge of experts and manipulate the expertise to solve problems at an expert s level of performance. KBS has four major components: knowledge base, inference engine, user interface, and explanation subsystem. Developing KBS involves capturing domain knowledge and knowledge about problem solving methodology regarding real world problems associated with the particular domain of application. Rule based reasoning (RBR) and case based reasoning (CBR) are the two fundamental and complementary reasoning methods of KBS (Pal and Campbell, 1997). KBS have widely been adopted to solve decision making problems in many domains (Chan et al., 2000; Kathuria et al., 1999; Vlahavas et al., 1999; McIvor and Humphreys, 2000; Liao, 2005; Tan et al., 2005; Stahl, 2006; Ricci et al., 2002). In this paper we propose hybrid knowledge based system (HKBS) approach to assists /$ see front matter 2011 Elsevier Inc. All rights reserved. doi: /j.jss

2 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) decision makers in evaluation and selection of the software packages. Rest of the paper is organized as follows: Section 2 provides review of literature in the field of software evaluation and selection. Section 3 describes decision making framework comprising: methodology for software selection, criteria for evaluating software packages, and HKBS approach for evaluation and selection of the software packages. Section 4 illustrates applicability of HKBS approach for evaluation and selection of the software packages and compares it with the widely used existing software evaluation techniques AHP and WSM. Evaluation of HKBS approach is discussed in Section 5. The paper is concluded in Section Literature review In recent years, researchers have focused on models and methods for reusable off-the-shelf software selection (Grau et al., 2004; Kizzort, 2001; Kontio et al., 1996; Kunda, 2003; Leung and Leung, 2002; Oh et al., 2003). However, there exists other literature which describes: evaluation and selection of specific software products such as CASE tool (Blanc and Korn, 1992; Plessis, 1993), simulation software (Cochran and Chen, 2005; Hlupic and Paul, 1996; Nikoukaran et al., 1998; Tewoldeberhan et al., 2002), DSS software (Blanc and Jelassi, 1989; Reimann and Waren, 1985), AHP software (Ossadnik and Lange, 1999), knowledge management tool (Ngai and Chan, 2005; Patel and Hlupic, 2002), data mining software (Collier et al., 1999), visual programming language (Kiper et al., 1997), ERP package (Illa et al., 2000), CRM package (Colombo and Francalanci, 2004), expert system shell (Stylianou et al., 1992), and operations management software (Shtub et al., 1988) methodology for software selection (Blanc and Korn, 1992; Hlupic and Paul, 1996) criteria for software selection (Arditi and Singh, 1991; Chau, 1995; Reed, 1982; Reimann and Waren, 1985; Stylianou et al., 1992) automated systems/tools that assist decision makers in various activities involved in evaluating and selecting software packages (Bandini et al., 2001; Grau et al., 2004; Hlupic and Mann, 1995; Kathuria et al., 1999; Mohamed et al., 2004; Vlahavas et al., 1999) evaluation of single software attribute, quality or some quality sub-attributes, for a software product (Franch and Carvallo, 2003) an empirical study aimed at comparing two tool supported requirements prioritization methods AHP and case-based ranking (CBRank). The comparison is based on three measures: the ease of use, the time consumption, and accuracy. The results showed that for the first two measures CBRank overcomes AHP, while for accuracy AHP performs better than CBRank, even if the resulting ranks from the two methods are very similar (Perini et al., 2009). framework for requirements prioritization, which adopts an elicitation process based on acquisition of pairwise preferences (Avesani et al., 2004). A summary of contribution of the reviewed literature in the field of evaluation and selection of the software packages is given in Table 1. The data extracted from the reviewed literature includes software evaluation criteria, methodologies for software selection, software evaluation techniques, and systems/tools to support decision makers in evaluating and selecting software packages. After reviewing the relevant literature, it seems that analytical hierarchy process (AHP) and weighted scoring method (WSM) have widely been used for evaluation of the software packages. Some recent studies described fuzzy based approach for software selection. Each technique has its own strengths and weaknesses. On the basis of practical applications of these techniques discussed in the literature and on our own experience/judgment we attempt to analyze strengths and weaknesses of these techniques. AHP enables decision makers to structure a decision making problem into a hierarchy, helping them to understand and simplify the problem. But it is time consuming technique because of the mathematical calculations and number of pair wise comparisons which increases as number of alternatives and criteria increases or changes. Ranking of the alternatives in AHP depends on alternatives considered for evaluation. Adding or deleting alternatives can lead to change in the final ranking (rank reversal problem). Weighted sum method is easy to use and understand but weights to the attribute are assigned arbitrary and it is difficult task when a number of criteria is high. Another problem with weighted scoring method is that common numerical scaling is required to obtain final score. In fuzzy based approach decision makers can use linguistic terms to evaluate alternatives that improves decision making procedure by accommodating the vagueness and ambiguity in human decision making. However, it is difficult to compute fuzzy appropriateness index values and ranking values for all alternatives. Evaluation techniques discussed above helps to rank the candidate software packages and choose the best one but they lack in indicating how well each candidate software package meet user needs of the package. Another problem with both AHP and WSM is that these techniques do not adopt requirement-driven approach and hence they are inadequate for software selection decision making (Ncube and Dean, 2002). KBS especially case-based reasoning technique has capability to indicate how well each candidate software package meet user needs of the package. Our other observations based on review of literature (Jadhav and Sonar, 2009) are: (1) there is a little work done on developing decision making framework comprising: methodology for selecting software packages, criteria for evaluating software packages, technique for evaluating software packages, (2) there is need of system/tool having inbuilt knowledge of software evaluation criteria and evaluation technique which will assist decision makers not only in software selection but also increase efficiency, and brings consistency and transparency in the process of software selection, (3) although, functional criteria for software selection are different for different software packages, other criteria related to the quality, cost and benefits, vendor, hardware and software requirements, opinion of different stakeholders about the software package, and output characteristics of the software package are common and can be used for evaluation of any software package. 3. Decision making framework for software selection In this section we describe decision making framework for evaluation and selection of the software packages which comprises: (i) methodology for selection of the software packages, (ii) common criteria for evaluating software packages, and (iii) HKBS approach for evaluation and selection of the software packages Methodology for selection of the software packages Methodology for selection of the software packages involves procedures and steps that decision maker follows while software selection decision making. Methodology is not intended as rigid structure, it is intended as guideline or aid that can be adapted according to requirements of the individual organization. On the basis of review of literature we have developed a generic stage based methodology for selection of the software packages which comprises six stages as follows:

3 1396 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) Table 1 A summary of contribution of the reviewed literature. Author(s) Software Evaluation criteria Selection methodology Evaluation technique Reed (1982) GIS software Yes No No Jarkee and Vassiliou (1985) Database query language Yes No Yes Reimann and Waren (1985) DSS software Yes No No Shtub et al. (1988) Operations management software Yes No Yes Toshtzar (1988) Computer software Yes No Yes Blanc and Jelassi (1989) DSS software Yes Yes Yes Arditi and Singh (1991) Software in constructing accounting Yes No No Blanc and Korn (1992) CASE tools Yes Yes No Stylianou et al. (1992) Expert system shell Yes Yes No Plessis (1993) CASE tool Yes No Yes Chau (1995) Software in small businesses Yes No No Hlupic and Paul (1996) Manufacturing simulation software Yes Yes No Kontio et al. (1996) COTS selection Yes Yes Yes Kiper et al. (1997) Visual programming language Yes No Yes Nikoukaran et al. (1998) Simulation software Yes No No Collier et al. (1999) Data mining software Yes Yes Yes Lai et al. (1999) Multimedia authoring system Yes No Yes Ossadnik and Lange (1999) AHP software Yes No Yes Illa et al. (2000) ERP Yes Yes No Patel and Hlupic (2002) Knowledge management tool Yes Yes No Tewoldeberhan et al. (2002) Discrete-event simulation Yes Yes Yes Adhikari et al. (2004) International accounting software Yes No No Colombo and Francalanci (2004) CRM packages Yes Yes Yes Cochran and Chen (2005) Object oriented simulation software Yes No Yes Ngai and Chan (2005) Knowledge management tools Yes No Yes Kitchenham (1996) Software engineering methods and tools Yes Yes Yes Ochs et al. (2001) COTS Assessment and Selection Yes Yes Yes 1. Requirement definition: The first and most important phase in software selection process is to identify functional and non functional requirements of the software. It must be accurate, complete and detailed because it is used to select the most appropriate software package. 2. Preliminary investigation of availability of software packages: This phase includes preliminary investigation of availability of software packages that might be suitable candidate, including high level investigation of major functionalities and features supported by the software package. Helpful resources for preliminary investigation could be web based resources including vendor s web sites, professional association catalogs and other third party reports. Deliverable of this phase is a list of available software packages that might be candidate for evaluation. 3. Short listing packages: Candidate software packages identified in the second phase that does not provide essential functionalities and features or does not work with existing hardware, operating system, data management software or network are eliminated in this phase. Criteria related to vendor or price of the software package can also be used for eliminating some of the candidate software packages. Deliverable of this phase is list of candidate software packages to be considered for detailed evaluation. 4. Establishing criteria for evaluation: In this phase criteria to be used for evaluation of the software packages are identified and arranged in hierarchical tree structure format. Each branch in the hierarchy ends into well defined and measurable basic attribute. Deliverable of this phase is set of criteria arranged in hierarchical tree structure format. 5. Evaluating software packages: In this phase metrics are defined and weights are assigned to each basic attribute in the criteria hierarchy. Rating is done against each basic criterion in hierarchy for each software package considered for detailed evaluation. Aggregate score is then calculated for each software package. 6. Selecting software package: The final phase is to rank the available alternatives in descending order of the score and select the best software. Aggregate scores give us only an idea about which one is better over the other; however, decision of selecting best software package, as in other selection, is always human dependable. Price/performance tradeoff can be considered to identify the software package which represents the best value. After selecting the best alternative, negotiate and sign contract with vendor of the selected software package by specifying software price, number of licenses, payment terms and conditions, functional specification, repair and maintenance responsibilities, time table for delivery, and options to terminate the agreement Software evaluation criteria Many literature concerning evaluation and selection of the software packages provides software evaluation criteria to evaluate specific software package such as Data Mining, CRM, and ERP. However, focus of these literatures is not on providing a generic list of criteria that could be used for evaluation of any software package. It has also been observed that meaning of each evaluation criterion is open to evaluators own interpretation and sometimes terminology used by one author for a criterion in one literature is different than terminology used by another author for the same criterion, which might confuse evaluator. Some literature describes only functional and quality attributes of the software packages and does not focus on other important criteria related to the vendor, cost and benefits, technical (hardware and software) requirements, opinion, and output related characteristics of the software package. In this section we provide taxonomy (generic list) of software evaluation criteria that could be used for evaluation of any software package. We also define meaning of each evaluation criterion along with its associated measures that are essential for assessment of the candidate software packages. The evaluation criteria are investigated on the basis of thorough literature review and are broadly categorized into seven different groups as shown in Fig Functional criteria Criteria related to the functional capabilities of the software package are different for different software packages, e.g. CRM, ERP, Data Mining, etc. Therefore, functional criteria can not be generalized hence those are not discussed here.

4 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) Table 2 Quality criteria. Sub-criteria Basic criteria Criteria meaning Metrics Portability Middleware standards Breadth of the middleware standards supported by the software package DBMS standards Breadth of the DBMS systems supported by the software package CORBA, DCOM, RMI, ODBC, JDBC, OLE DB MS-Access, MS-SQL, MS-Excel, Oracle, DB2, Informix, Sybase, MySQL, Ingrace EDI, XML Communication standards Inter-organizational data exchange standards supported by the software package OS compatibility Package compatibility with the operating systems MS-Windows, Novell, Unix, Linux, MAC Hardware compatibility Package compatibility with the hardware Supercomputer, Mainframe, Minicomputer, Microcomputer Personalizability Vertical solution Number of customized versions of the software package Numeric Customizable fields Ability to personalize layout of the software package Yes, No Customizable reports Ability to personalize layout of the report produced by the Yes, No software package Interface type Interface type of the software package Character based, Window based Programming languages Ability to personalize modules of the software package by Yes, No programming languages Maintainability Modules Average size of the independent code modules Numeric Number of independently Level of independence among the modules by indicating Numeric installable modules whether groups of modules or sub-modules need to be simultaneously installed even if only subset of them is required Number of workstations Maximum number of simultaneous users that can be Numeric supported by the software package Maximum number of distribution Ability to split the software package into separate application Numeric tiers tiers that can be distributed onto different servers Number of modules that can be Ability to distribute modules on different servers Numeric installed on separate servers Scalability Ability to support an increasing number of users and higher Yes, No load of transaction Usability User interface Ease with which user can use interface of the software package Yes, No Learning curve Ease with which user can learn and operate the package Yes, No User types Ability of the software package to support beginners, intermediate, advanced users or a combination of user types Beginners, intermediate, advanced users, and experts Data visualization Capability of the software package to present data effectively Yes, No Error reporting Error reporting and messaging ability of the software package Yes, No Domain variety Capability of the software package to be used in different Yes, No industries to solve different kinds of business problems Reliability Robustness Capability of the software package to run consistently without crashing Very poor, poor, fair,, very Backup and recovery Capability of the software package to support backup and Yes, No recovery feature Efficiency Time behavior Capability of the software package to produce results in reasonable amount of time relative to data size Very poor, poor, fair,, very Security Auditing Products logging and auditing capabilities Very poor, poor, fair,, very Password Package support for password Yes, No Data security Level of support for data security Very poor, poor, fair,, very Individual and group access rights Package support for managing and enforcing access rights Yes, No Field level security Package support for security at the field level Yes, No Data/document encryption Package support for data/document encryption Yes, No Functional Software evaluation criteria Technical Quality Vendor Output Fig. 1. Software evaluation criteria. Cost and benefit Opinion Quality criteria Quality criteria are used to assess quality of the software package. We have used ISO/IEC 9126 quality model as a raw basis to refine, customize and to form our quality model which can be used for evaluation and selection of any software package. According to ISO/IEC quality model there are six major quality characteristics namely functionality, reliability, usability, efficiency, maintainability and portability. Functionality of the software package is nothing but ability of the software package to perform according to specific needs of the organization and it is used to measure the level to which software package satisfies functional requirements of the organization. As mentioned earlier functional criteria are different for different software packages therefore we have moved it from quality criteria group and formed separate functional criteria category. Portability is ability of the software product to be transferred from one environment to another. Maintainability is the ability of the software to be modified. Modification can include corrections,

5 1398 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) Table 3 Vendor criteria. Sub-criteria Basic criteria Criteria meaning Metrics Training and documentation criteria Tutorial Availability of tutorial to learn how to use the software Yes, No package Troubleshooting guide Availability of troubleshooting guide Yes, No Training Vendor support for training courses to learn the Yes, No software package User manual Availability of user manual with indexes, with Yes, No important information and the main commands Maintenance and up-gradation Consultation Availability of technical support and consultancy by Yes, No the vendor Communication Communication with the vendor Telephone, 24 7 toll free, online Demo Availability of on-site demo and free-trial version Yes, No Response time Level of service rendered by the vendor 24 h, 3 days, 1 week Business skills Business skills of the vendor Very poor, poor, fair,, very Vendor reputation Vendor popularity Popularity of vendor in the market Very poor, poor, fair,, very Product history Popularity of vendor product in the market Very poor, poor, fair,, very Length of experience Experience of vendor about development of the software package Very poor (less than 1 year), poor (1 2 years), fair (3 5 years), (6 8 years), very (above 9 years) Number of installations Number of installations of the software package Very poor (less than 3), poor (4 10), fair (11 20), (21 30), very (above 30) Number of references Number of references of the existing customers Very poor (less than 3), poor (4 10), fair (11 20), (21 30), very (above 30) Past business experience Past business experience with the vendor, if any Very poor, poor, fair,, very improvements or adaptations of the software to adjust to changes in the environment, in terms of functional requirements and specifications. Usability is understandability of the software packages as well as the easiness to learn and operate it under certain specific condition. Reliability is ability of the software package to run consistently without crashing under specific condition. Efficiency is ability of the software package to provide appropriate performance, relative to the amount of resources used under certain conditions. Personalizability is ability of the software package to be customized according to the user needs. Quality criteria are decomposed further into sub-criteria and these one into measurable attributes as described in Table Vendor criteria Vendor criteria are used to assess vendor capabilities of the software packages. These are decomposed further into sub-criteria and these one into measurable attributes as described in Table Cost and benefits criteria These criteria are used to assess cost and benefits related characteristics of the software package. Measurable attributes of the cost and benefits criteria are described in Table Opinion criteria Measurable attributes of criteria related to the opinion of different stakeholders of the software package are described in Table Technical criteria Measurable attributes of criteria related to the technical requirements (hardware and software) of the software package are described in Table Output criteria Measurable attributes of output related characteristics of the software packages are described in Table Software evaluation: a HKBS approach In this section we describe HKBS approach that employs an integrated rule based and case based reasoning techniques for evaluation and selection of the software packages. Rule based reasoning, a deductive reasoning approach, is reasoning system which mimics problem solving behavior of human experts. Knowledge base of rule based system comprises knowledge that is specific to domain of application. Rule based reasoning is appropriate for problem solving tasks that are well constructed. However, it does not make substantial use of the knowledge embedded in previous cases. Case based reasoning, an inductive reasoning approach, is problem solving approach that solves problem by adapting solution of more similar case that has been solved in the past. A new problem is matched with cases stored in case base of the system and one or more similar cases are retrieved. Solution suggested by most matching case is then reused and tested for success (Aamodt and Plaza, 1994). Integration of rule based and case based reasoning technique eliminates drawbacks of both pure inductive and deductive reasoning and solves problem in more intelligent way. The architecture of HKBS is shown in Fig. 2. The HKBS is developed using intelligent system development and solution framework proposed by Sonar (2004, 2007). HKBS employs an integrated rule based and case based reasoning approach for evaluation and selection of the software packages. Rule base reasoning component of the HKBS stores domain knowledge, i.e. knowledge about software evaluation criteria which is used to: (i) assist decision makers in choosing evaluation criteria which he/she wish to consider for evaluating software packages, (ii) capture users requirements of the software package through a simple or knowledge driven sequence of the forms and formulate a problem case, and (iii) control execution of CBR. Once user needs of the software package are captured next step is submit these requirements as input to the CBR component of the HKBS. Case based reasoning

6 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) Table 4 Cost and benefits criteria. Basic criteria Criteria meaning Metrics License cost License cost of the software package Numeric Hardware and software cost Cost of an additional hardware and software required to Numeric run the software Installation and implementation cost Installation and implementation cost of the software Numeric package Maintenance cost Maintenance cost of the software package Numeric Training cost Training cost of the software package Numeric Upgrading cost Upgrading cost of the software package Numeric Direct benefits Direct benefits of the software package Very poor, poor, fair,, very Indirect benefits Indirect benefits of the software package Very poor, poor, fair,, very Table 5 Opinion criteria. Basic criteria Criteria meaning Metrics End users Opinion of end users about the software package Very poor, poor, fair,, very External consultants Opinion of external consultants about the software package Very poor, poor, fair,, very In-house experts Opinion of in-house experts about the software package Very poor, poor, fair,, very Magazines Opinion about the software package given in the magazines Very poor, poor, fair,, very Outside personal acquaintances Opinion of outside personnel about the software package Very poor, poor, fair,, very Product leaflets Opinion about the software package in product leaflets Very poor, poor, fair,, very Subordinates Opinion of subordinates about the software package Very poor, poor, fair,, very Vendor and sales representatives Opinion of vendor and sales representatives about the software package Very poor, poor, fair,, very technique is then used to: (i) compare user needs of the software package with description of the candidate software packages stored as cases in case base of the system and (ii) retrieve software packages which most closely meet user requirements of the software package and rank them according to similarity score. Similarity score indicates how well each candidate software package meet user needs of the software package. Case base of the system is made up of detailed description of the candidate software packages to be evaluated. Each software package is described using well defined set of features and feature values. Features used to describe software package have same vocabulary as that of criteria used for evaluation of the software packages. 4. Comparison of AHP, WSM and HKBS The review of literature shows that AHP has widely been used for evaluation and selection of the software packages. WSM is another common approach that has been used for evaluation and selection of the software packages. As this study propose HKBS approach for evaluation and selection of the software packages there is need to do comparison of AHP, WSM and HKBS approach for software selection. Therefore, aim of this section is to study and compare these evaluation techniques by applying them for evaluation and selection of the software components. The data about the software components to be evaluated is taken from the study A quality framework for developing and evaluating Fig. 2. HKBS approach for evaluation and selection of the software packages.

7 1400 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) Table 6 Technical criteria. Basic criteria Criteria meaning Metrics Communication protocol Communication protocols supported by the package TCP/IP, UDP, NETBUI, HTTP, FTP, SOAP, etc. External storage External storage capacity required Numeric Network technology Network technology supported by the package LAN, WAN, MAN Primary storage Primary storage capacity required Numeric original software components (Andreou and Tziakouris 2007). The study proposed a quality framework for developing and evaluating original software components. The framework is demonstrated and validated by applying it in searching a two-way SMS messaging component to be incorporated in an online trading platform. The reasons behind using data from this study are (i) the case presented in the study represents real world situation, (ii) the data and user needs of the software component are described in detail, (iii) results of evaluation of the software components are also discussed in the study, and (iv) hierarchy of the criteria used for evaluation of the software components is similar to the hierarchy of the criteria for evaluation and selection of the software packages. The four components considered for evaluation are: ActiveSMS, SMS- Demon, GSMActive, and SMSZyneo. Table 8 provides features of these software components in detail and Table Evaluation using AHP AHP was proposed by Dr. Thomas Saaty in late 1970s and has been applied in wide variety of applications in various fields. This method allows consideration of both objective and subjective factors in selecting the best alternative. The methodology is based on three principles: decomposition, comparative judgments and synthesis of priorities. Decomposition principle calls for construction of hierarchical network to represent a decision problem with top level representing overall objectives (goal) and lower levels representing criteria, sub-criteria and alternatives. With the comparative judgments users are required to set up a comparison matrix at each level of hierarchy by comparing pairs of criteria or sub-criteria. In general comparison takes the form: How important is criteria C i relative to criteria C j? Questions of this type are used to establish weights of criteria. The possible judgments used for pairwise comparison and their respective numerical values are described in Table 10. Similar questions are to be answered to access the performance scores of alternatives on the subjective (judgmental) criteria. Let A ij denotes the value obtained by comparing alternative Table 7 Output criteria. Basic criteria Criteria meaning Metrics Printer Package support for printer output Yes, No File Package support for file output Yes, No Other software Package support for output to other software Yes, No A i to alternative A j relative to the criterion C i. As decision maker is assumed to be consistent in making judgments about any one pair of alternative and since all alternatives will always rank equally when compared to themselves, we have A ij =1/A ji and A ii = 1. This means that it is only necessary to make 1/2 * m *(m 1) comparisons to establish full set of pairwise judgments (m represents number of alternatives). The final stage is to calculate aggregate performance value for each alternative and ranking of alternatives according to their performance. Aggregate score is obtained using the following formula: provides details of the evaluation criteria (features), its weight (importance), metrics, and user requirements of the software component. R i = W k A ik where R i is overall score of ith alternative, W k is importance (weight) of kth criterion, and A ik is relative score of ith alternative with respect to kth criterion. As mentioned above the first stage in AHP is formulating decision hierarchy. The decision hierarchy formulated for selection of the software component is depicted in Fig. 3. The highest level of hierarchy represents the goal, second level represents the criteria, third level represents the sub-criteria and fourth level represents the software components to be evaluated. The second stage in AHP is obtaining pairwise comparison matrix and normalized matrix by comparing each alternative over the others with regard to each evaluation criterion. The judgements Goal Selecting the best suitable software components Criteria Functionality Reliability Efficiency Maintainability Sub-criteria User Satisfaction Service Satisfaction Error prone Correctness Throughput Upgradeability Backward Compatibility Access control Capacity Alternatives ActiveSMS SMSDemon GSMActive SMSZyneo Fig. 3. Decision hierarchy for the component selection.

8 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) Table 8 Details of the software components. Evaluation criteria ActiveSMS SMSDemon GSMActive SMSZyneo User satisfaction Service satisfaction 9/12 5/12 8/12 6/12 Access control Provided Provided Provided Provided Error prone 0/day 1/day 0/day 0/day Correctness Throughput 60/min 8/min 120/min 8/min Capacity Upgradeability Backward compatibility Provided Provided Provided Provided Source: Andreou and Tziakouris (2007). Table 9 Details of the software component evaluation criteria. Criteria Sub-criteria Weight (%) Metrics User requirements Functionality User satisfaction 20 Level of satisfaction on the scale of Service satisfaction 20 Functions Ratio 12 Access control 5 Provided or not Provided Reliability Error prone 10 Number of errors/crashes per unit of time 0 Correctness 10 Ratio of successful SMS sending 1 Efficiency Throughput 15 Number of requests per unit of time 50 Capacity 10 Number of GSM modems supported 5 Maintainability Upgradeability 5 Level of satisfaction on the scale of Backward compatibility 5 Provided or Not Provided Source: Andreou and Tziakouris (2007). for pairwise comparison have been done by the authors by considering user requirements (see Table 9) and software components specification (see Table 8). For example, the pairwise comparison matrix and normalized matrix with respect to user satisfaction and service satisfaction criteria is shown in the Tables Similarly normalized score is obtained for each alternative with regard to each evaluation criterion. The normalized score of each alternative with regard to each evaluation criterion is shown in Table 15. Table 10 Pairwise comparison judgments between elements X and element Y. Judgment Values X is equally preferred to Y 1 X is moderately preferred over Y 3 X is strongly preferred over Y 5 X is very strongly preferred over Y 7 X is extremely preferred over Y 9 Intermediate values; when compromise is needed 2, 4, 6, 8 Preference of Y compared to X 1/2, 1/3, 1/4, 1/5, 1/6, 1/7, 1/8, 1/9 Table 11 Pairwise comparison matrix with respect to user satisfaction. ActiveSMS SMSDemon GSMActive SMSZyneo ActiveSMS SMSDemon 1/5 1 1/5 3 GSMActive SMSZyneo 1/8 1/3 1/8 1 Table 12 Normalized alternative score with respect to user satisfaction. ActiveSMS SMSDemon GSMActive SMSZyneo Average ActiveSMS SMSDemon GSMActive SMSZyneo Table 13 Pairwise comparison matrix with respect to service satisfaction. ActiveSMS SMSDemon GSMActive SMSZyneo ActiveSMS SMSDemon 1/4 1 1/3 1/2 GSMActive 1/ SMSZyneo 1/3 2 1/2 1 Table 14 Normalized alternative score with respect to service satisfaction. ActiveSMS SMSDemon GSMActive SMSZyneo Average ActiveSMS SMSDemon GSMActive SMSZyneo The third stage in AHP is identifying preferred alternative by calculating aggregate score of each alternative. Aggregate score is calculated by multiplying normalized score with the weight (importance) of that criterion and sum the results for all criteria. The preferred alternative will have the highest score. The calculation of aggregate score for each alternative using AHP is shown in Table 15. The illustration of evaluation of the software components using AHP shows that large number of pairwise comparisons are required to be done and it depends on (i) number of alternatives to be evaluated and (ii) number of criteria considered for evaluation. The formula for calculating number of pairwise comparisons in AHP is as follows: Number of pairwise comparisons = M M 1 N 2 where M is number of alternatives and N is number of criteria considered for evaluation. In an example of evaluation of the software components, number of alternatives and criteria considered for evaluation are 4 and

9 1402 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) Table 15 Calculations of aggregate score of each software component using AHP. Component Criteria Weight Normalized score ActiveSMS User satisfaction Service satisfaction Access control Error prone Correctness Throughput Capacity Upgradeability Backward compatibility Total score SMSDemon User satisfaction Service satisfaction Access control Error prone Correctness Throughput Capacity Upgradeability Backward compatibility Total score GSMActive User satisfaction Service satisfaction Access control Error prone Correctness Throughput Capacity Upgradeability Backward compatibility Total score SMSZyneo User satisfaction Service satisfaction Access control Error prone Correctness Throughput Capacity Upgradeability Backward compatibility Total score Score 9, respectively. Therefore the total number of pairwise comparisons done is calculated as follows: Total number of pairwise comparisons = = 54 2 If there are 10 alternatives to be evaluated using 100 criteria, then total number of pairwise comparisons required to be done are (1/2 * (10 * (10 1)) * 100) = Even though there are tools available to do mathematical calculations pairwise comparisons must be done by experts and it is tedious and time consuming job. If user requirements of the software package changes then relative score of each alternative over the other with respect to each evaluation criterion also changes. If number of alternatives considered for evaluation increases then number of pairwise comparisons also increases. Therefore in both the cases: (i) if user needs of the package changes, or (ii) number of alternatives to be evaluated changes, pairwise comparison must be done again which is tedious and time consuming task Evaluation using WSM Consider m alternatives {A 1, A 2,..., A m } with n deterministic criteria {C 1, C 2,..., C n }. The alternatives are fully characterized by decision matrix {S ij }, where S ij is score that measures how well alternative A i performs on criterion C j. The weights {W 1, W 2,..., W k } accounts for relative importance of the criteria. The best alter- Table 16 Calculations of aggregate score of each software component using WSM. Component Criteria Weight Rating Score ActiveSMS User satisfaction Service satisfaction Access control Error prone Correctness Throughput Capacity Upgradeability Backward compatibility Total score 440 SMSDemon User satisfaction Service satisfaction Access control Error prone Correctness Throughput Capacity Upgradeability Backward compatibility Total score 215 GSMActive User satisfaction Service satisfaction Access control Error prone Correctness Throughput Capacity Upgradeability Backward compatibility Total score 420 SMSZyneo User satisfaction Service satisfaction Access control Error prone Correctness Throughput Capacity Upgradeability Backward compatibility Total score 225 native is one with the highest score. In WSM the final score for alternative A i is calculated using the following formula: S(A i ) = W j S ij where sum is over j =1, 2,..., n; W j is relative importance of jth criterion; S ij is score that measures how well alternative A i performs on criterion C j. WSM works only with the numeric data. Therefore, rating for each alternative must be done with regard to each evaluation criterion. In an example of component selection except user satisfaction and upgradeability criteria, rating is not available for the other criteria. Therefore, all alternatives have been first rated with regard to each evaluation criteria by considering user requirements and component support for that particular feature. The rating and calculations of aggregate score for each software component using WSM is shown in Table Evaluation using HKBS HKBS is an integration of rule based and case based reasoning techniques. RBR component of HKBS assists decision makers to: (i) select criteria for evaluation of the software components, (ii) capture users requirements of the software component through simple or knowledge driven sequence of forms, and (iii) formulating a problem case. An example of how system assists decision makers in choosing evaluation criteria and specifying user needs of the software component is shown in Figs. 4 and 5, respectively.

10 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) Fig. 4. Interface for selecting software component evaluation criteria. Fig. 5. Interface for specifying user needs of the software component. Fig. 5 provides interface to specify user requirements (ideal values) for upgradeability and backward compatibility criteria. As the scale of measurement for upgradeability criterion is level of satisfaction on the scale of 1 (very poor) to 5 (very ), the obvious choice for the ideal value is 5 (very ). In case of backward compatibility criterion, user requirement is software components should support backward compatibility therefore ideal value is yes. Similarity functions used to calculate feature and case level similarities have been discussed in the later part of this section. Rules RBR are written in simple IF...THEN...ELSE format. An example of one of the rules written to suggest criteria for evaluation of the portability characteristics, and capture user requirements of portability characteristics of the software component is given below. Rule No = 1 ID = SoftwareSelection.Portability Criteria IF SoftwareSelection.MainTask IS Software Evaluation AND SoftwareSelection.MainCriteriaCategory INCLUDES ANY OF [Quality] AND SoftwareSelection.QualityCriteriaCategory INCLUDES ANY OF [Portability] AND ASK(SoftwareSelection.PortabilityCriteria) AND ASK(SoftwareSelection.ShowForm PortabilityCriteria) THEN SoftwareSelection.Final Goal IS Portability Criteria Most important component of the CBR system is case schema which is collection of case features. Each case feature is linked to similarity function used to calculate similarity between problem case and solution case feature. In this study problem case is nothing but user needs of the software component and solution cases are nothing but candidate software components to be evaluated. The similarity knowledge which is stored in the form of case schema is used to assess similarity between software component and user needs of that component. CBR component of the HKBS is used to: (i) retrieve candidate software components stored as cases in case base of the system, (ii) compare user requirements of the software component with description of the candidate software components and calculate similarity score, and (iii) rank the components in descending order of the similarity score. The similarity score indicates how well each candidate software component meets user requirements of the software component. In CBR system, the quality of the results mainly depends on the similarity measures that are used to retrieve similar cases. During the retrieval process, the current problem is matched against cases in the case base. Matching is the process of comparing two cases and determining their degree of similarity. If the case is represented as set of features and feature values, Kolodner (1993) suggests following formula to calculate case level similarity: n Wi Sim(qv,cv) i=1 Sim(P, S) = n i=1 Wi where Sim(P, S) represents case level (global) similarity; Wi is relative importance (weight) of the feature in similarity assessment process; and Sim(qv, cv) is individual feature level (local) similarity between query value qv and case value cv of the feature. In HKBS, cases are represented as a set of feature and feature values. Therefore, we have used case level similarity function Sim(P, S), as suggested by Kolodner (1993), for calculation of the case level similarity. The individual feature level similarity is calculated using local similarity function linked to that feature and it depends on type of the feature. Local similarity for numeric type of features is calculated using the functions given below (Avramenko and Kraslawski, 2006). These similarity measures have also been used for online product recommendation systems developed using ikenstudio [ (Godse et al., 2009). Sim(qv,cv) = 1 d(qv,cv) range(v) d(qv,cv) = qv cv

11 1404 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) range(v) = Max v Min v where qv: query value of the feature cv: case value of the feature Max v: maximum value which feature can take Min v: minimum value which feature can take Other functions that have been used in HKBS to calculate individual feature level similarity for numeric type of features are as follows. The similarity functions MaxValueSame and MinValue- Same have been used for similarity calculations of higher is better and lower is better type of features respectively. These similarity functions have also been used in a case based recommendation of product catalog of operational amplifiers (Bergmann, 2003). MaxValueSame: Sim(qv,cv) = 1 if(qv cv) = Numeric sim(qv,cv) MinValueSame: Sim(qv,cv) = 1 if(cv qv) = Numeric sim(qv,cv) For qualitative (symbolic) type of features similarity is calculated using the function given below. Sim(qv,cv) = S(qv,cv) where S(qv,cv) is similarity defined explicitly between query value (qv) and case value (cv) of the feature. The evaluation result (ranking) of the software components produced by HKBS is shown in Fig. 6. Functional and quality criteria column indicates how well each component meets functional and quality requirements of the software component respectively. The case matching column indicates how well each software component meets overall (functional and quality requirements) requirements of the software component. If number of criteria (features) used for evaluation are less than the number of criteria (features) in case schema, then criteria matching column is used to check how well each candidate software component meets user requirements. If all criteria of the case schema are used for evaluation of the software components then case matching and criteria matching values are same. The result shows that ActiveSMS component is better than the other software components. The evaluation result (ranking) of software components obtained using HKBS is similar to the result (ranking) obtained using AHP and WSM Comparison of AHP, WSM and HKBS The comparison of AHP, WSM, and HKBS has been done on the basis of following parameters and summarized in Table 17: Does evaluation technique supports qualitative criteria? Does evaluation technique supports quantitative criteria? Is evaluation technique simple and easy to use when (i) number of alternatives to be evaluated and number of criteria considered for evaluation are large in number, (ii) number of alternatives to be evaluated changes, (iii) number of evaluation criteria changes, and (iv) user needs of the software package changes Does evaluation technique support knowledge/experience reuse? Does evaluation technique support to capture/change user requirements of the software package? Does evaluation technique cause rank reversal (reversal in ranking) problem, if new alternative is added? Does evaluation technique produce result indicating how well each alternative meet user requirements? The comparison of AHP, WSM and HKBS for evaluation and selection of the software components shows that HKBS approach is comparatively better than AHP and WSM with regard to following aspects: Computational efficiency HKBS works well with both qualitative as well as quantitative criteria HKBS approach for software evaluation is comparatively easy to use when: Number of evaluation criteria or number of alternatives to be evaluated is large in number. For example, if 10 alternatives are to be evaluated using 10 evaluation criteria using AHP (provided importance of each criteria is given) then total number of pairwise comparisons required to be done are 450 which is quite tedious and time consuming task. Number of alternatives to be evaluated changes. For example, in the case of software component selection using AHP, if number of components to be evaluated increases from 4 to 10, the number of pairwise comparison also increases from 54 to 405 which is quite tedious and time consuming task. Criteria to be considered for evaluation changes. Requirements of the software package changes Knowledge reuse HKBS retains knowledge about software evaluation criteria and similarity knowledge to determine the fit between software package and user requirements of the package. This knowledge can Fig. 6. Result of evaluation of the software components using HKBS.

12 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) Table 17 Comparison of AHP, WSM and HKBS. Parameters Evaluation techniques AHP WSM HKBS Support for qualitative criteria Yes No Yes Support for quantitative criteria Yes Yes Yes If number of alternatives to be evaluated increases Pairwise comparisons also increases and needs to be done to calculate final score If number of evaluation criteria changes If user requirements changes Pairwise comparisons needs to be done again to calculate final score Pairwise comparison needs to be done again to calculate final score Rating of each newly added alternative with regard to each evaluation criterion must be done to calculate final score No extra efforts are required to calculate final score except rating to be done for newly added criteria Rating of each alternative with regard to each evaluation criteria changes and needs to be done again Any number of alternatives can be added or removed with no special efforts required except adding a new case (alternative description) in case base No special efforts are required to calculate similarity score If evaluation criteria already exists in the system, otherwise that criterion needs to be included in the system along with similarity function Provides flexibility to change user requirements and calculate similarity score accordingly, with no special efforts required Support for Knowledge/experience reuse No No Yes (as HKBS uses CBR) Support to specify and change user requirements No No Yes (as HKBS is a tool) Rank reversal problem Yes No No Support to indicate how well each alternative meet user needs of the software package No No Yes be reused later for evaluation of the same or different software packages with different requirements of the same or different organizations Consistency and presentations of the Results Resulting score of AHP and WSM indicates relative ranking of the alternatives whereas result of HKBS shows not only ranking of the alternatives but also indicates how well each alternative meet user requirements of that package. In case of AHP and WSM aggregate score of each alternative may not remain consistent even though requirements are same because aggregate score depends on expert s own judgment which may not remain consistent for all the time. Whereas HKBS produce same results unless user requirements changes. In case of AHP adding an alternative may cause a rank reversal (reversal in ranking) problem that does not happens in case of HKBS Flexibility in solving software selection decision making problem HKBS provides flexibility to decision makers not only in choosing evaluation criteria but also for specifying and changing user requirements of the software package. Addition or deletion of the software packages in HKBS is easy as it uses case base for storing details of the candidate software packages. 5. Evaluation of HKBS approach This section presents results of evaluation of HKBS. After developing and applying HKBS for software selection, next step was to perform usability test to verify functionality, efficiency, effectiveness and convenience of the HKBS approach for software selection. To perform usability test we followed the approach used by Ricci and Nguyen (2007) for testing usability of the mobile phone recommender system. Three experts (testers) having knowledge/experience of evaluation and selection of the software packages were involved in the experiment so that they could provide us proper feedback on usability of the HKBS for software selection. The experiment comprised three phases: Training Testing Evaluation During training phase HKBS was introduced to the testers. Testers were trained on how system works and assists them in (i) choosing criteria for evaluation of the software packages and (ii) specifying user needs of the software package. Training phase typically lasted for min. During testing phase testers were asked to (i) select only those criteria/features which he/she wish to consider for evaluation of the software packages and (ii) specify user needs of the software package using selected criteria/features. It was also asked them to check results (ranking of the candidate software packages) of the system by changing user requirements of the software package and importance (weight) of the evaluation criteria. During evaluation phase testers evaluated performance of the HKBS by completing usability questionnaire. The questionnaire contained list of 10 questions and free-text space for the comments. Testers answered each question using a seven point likert scale where 1 is strongly agree and 7 is strongly disagree. Table 18 shows testers average rating for the questionnaire statement which expressed testers subjective evaluation of the systems performance. In particular, testers found that system effectively helped them in: choosing criteria for evaluation of the software packages specifying user needs of the software package determining the fit between software package and user needs of that package Testers also found that interaction with the system and user interfaces were pleasant and friendly. All testers explicitly mentioned that it is easy to specify and change user needs of the software package using HKBS. It assists decision makers to determine the fit between software package and user needs of that

13 1406 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) Table 18 Results of evaluation of HKBS. Statement Average rating Knowledge/experience reuse Flexibility in problem solving Consistency and presentation of the evaluation results. 1. This system effectively helped me in selecting criteria for 2.33 evaluation of the software packages 2. Evaluation criteria used in this system are enough to 1.33 evaluate and rank the candidate software packages 3. This system has used proper hierarchy of the evaluation 2.33 criteria 4. This system effectively helped me to specify user needs of 2.33 the software package 5. I can effectively complete task of evaluation and selection of 1.66 the software package using this system 6. I can efficiently complete task of evaluation and selection of 1.66 the software package using this system 7. Results, i.e. ranking of the candidate software packages, 2.66 produced by the system in the form of case matching is easy to understand 8. Results produced by the system effectively helped me in 2.00 determining the fit between software package and user needs of the package 9. This system has all the functions and capabilities I expected 3.00 it to have 10. Overall, I am satisfied with this system 2.66 package. They also mentioned that the results (ranking of candidate software packages) produced by the system in the form of percentage case matching (percentage similarity between user needs of the software package and capabilities of each candidate software package) are impressive. 6. Conclusions Evaluation and selection of the software packages is tedious and time consuming task. Issues such as identifying software evaluation criteria, establishing criteria hierarchy, defining its associated measures and application of sound decision aid methodology arise frequently not only for non experienced people but also for the skilled one. This study provides conceptual understanding of all aspects related to the software selection such as (i) methodology describing factors and issues that needs to be taken into consideration during the process of software selection and deliverables at each stage, (ii) software evaluation criteria along with its meaning and associated measures that are essential for assessment of the candidate software packages, and (iii) software evaluation techniques. Decision makers need to follow certain procedure during the process of software selection. This study proposed a stage-based methodology which decision makers can follow as guideline or aid for software selection. Identifying criteria for evaluation of the software packages is tedious and time consuming job for the decision makers. This study proposed a comprehensive list (taxonomy) of software evaluation criteria that are common and can be used for evaluation of any software package. Meaning of each evaluation criteria is also defined along with its associated measures. HKBS proposed in this study can be used by decision makers as a tool for software selection as it supports various software evaluation activities such as: choosing criteria for software evaluation; specifying and changing user requirements of the software package; determining the fit between software package and user needs of that package; and for reusing knowledge/experience. Applicability of HKBS for software component selection and comparison of AHP, WSM and HKBS shows that HKBS comparatively better than AHP and WSM with regard to following aspects: Computational efficiency Although views of the limited number of experts have been taken into consideration, usability test performed to verify functionality, efficiency, effectiveness and convenience of the HKBS shows that the HKBS effectively helps decision makers in choosing criteria for evaluation of the software packages, specifying and changing user requirements of the software package and determining the fit between software package and user needs of that package. References Aamodt, A., Plaza, E., Case based reasoning foundational issues, methodical variations and system approaches. AI Communications 7 (1), Adhikari, A., Lebow, M.I., Zhang, H., Firm characteristics and selection of international accounting software. Journal of International Accounting, Auditing and Taxation 13, Andreou, A.S., Tziakouris, M., A quality framework for developing and evaluating original software components. Information and software technology 49, Arditi, D., Singh, S., Selection criteria for commercially available software in constructing accounting. Project Management 9 (1), Avesani, P., Bazzanella, C., Perini, A., Susi, A., Supporting the requirements prioritization process. A machine learning approach. In: Proceedings of 16th International Conference on Software Engineering and Knowledge Engineering, KSI Press, Banff, Alberta, Canada, pp Avramenko, Y., Kraslawski, A., Similarity concept for case based design in process engineering. Computers and Chemical Engineering 30, Bandini, S., Paoli, F., Manzoni, S., Mereghetti, P., A support system to COTSbased software development for business services. In: SEKE 02 ACM, pp Bergmann, R. (Ed.), INRECA Methodology., 2nd edition. LNAI 1612, pp Blanc, L.A., Korn, W.M., A structured approach to the evaluation and selection of CASE tools. In: ACM. Blanc, L., Jelassi, M., DSS software selection: a multiple criteria decision methodology. Information and Management 17, Chau, P.Y.K., Factors used in the selection of packaged software in small businesses: views of owners and managers. Information and Management 29, Chan, C.W., Chen, L., Geng, L., Knowledge engineering for an Intelligent Case Based System for help desk operations. Expert Systems with Applications 18, Cochran, J.K., Chen, H., Fuzzy multi-criteria selection of object-oriented simulation software for production system analysis. Computers and Operations Research 32, Collier, K., Carey, B., Sautter, D., Marjanierni, C., A methodology for evaluating and selecting data mining software. In: Proceedings of 32nd Hawaii International Conference on System Sciences, pp Colombo, E., Francalanci, C., Selecting CRM packages based on architectural, functional, and cost requirements: empirical validation of a hierarchical ranking model. Requirements Engineering 9, Franch, X., Carvallo, J.P., Using Quality Models in Software Package Selection. IEEE Computer Society, pp Godse, M., Sonar, R., Jadhav, A., A hybrid approach for knowledge-based product recommendation. In: Third International Conference on Information Systems Technology and Management March 12 13, 2009, Communications in Computer and Information Science, Springer, vol. 31, pp Grau, G., Pablo Carvallo, J., Franch, X., Quer, C., DesCOTS: a software system for selecting COTS components. In: Proceedings of 30th EUROMICRO Conference, IEEE. Hlupic, V., Paul, R.J., Methodological approach to manufacturing simulation software selection. Computer Integrated Manufacturing Systems 9 (1), Hlupic, V., Mann, A.S., SimSelect: a system for simulation software selection. In: Proceedings of the 1995 Winter Simulation Conference, pp Illa, X.B., Franch, X., Pastor, J.A., Formalising ERP selection criteria. In: Proceedings of tenth international workshop on software specifications and design, IEEE. Jadhav, A.S., Sonar, R.M., Evaluating and selecting software packages: a review. Information and Software Technology 51, Jarkee, M., Vassiliou, Y., A framework for choosing database query language. Computing Surveys 17 (3), Kathuria, R., Anandarajan, M., Igbaria, M., Selecting IT applications in manufacturing: a KBS approach. Omega 27, Kiper, J.D., Howard, E., Ames, C., Criteria of evaluation of visual programming languages. Journal of Visual Languages and Computing 8, Kitchenham, B.A., Evaluating software engineering methods and tools Parts 5 to 8. ACM SIGSOFT Notes, Starting January Kizzort, B., Selection of components for OTS component based systems. IEEE.

14 A.S. Jadhav, R.M. Sonar / The Journal of Systems and Software 84 (2011) Kontio, J., Caldiera, G., Basili, V.R., Defining factors goals and criteria for reusable component evaluation. In: CASCON 96 Conference, Toronto, Canada, November Kolodner, J., Case Based Reasoning. Morgan Kaufmann, San Mateo, CA. Kunda, D., STACE: social technical approach to COTS software evaluation. In: Component Based Software Quality. Springer-Verlag, LNCS 2693, pp Lai, V.S., Trueblood, R.P., Wong, B.K., Software selection: a case study of the application of the analytical hierarchical process to the selection of a multimedia authoring system. Information and Management 36, Leung, K.R.P.H., Leung, Hareton K.N., On the efficiency of domain-based COTS product selection method. Information and Software Technology 44, Liao, S., Expert system methodologies and applications a decade review from 1995 to Expert Systems with Applications 28, Lin, H.-Y., Hsu, P.-Y., Sheen, G.-J., A fuzzy-based decision making procedure for data warehouse system selection. Expert Systems with Applications. McIvor, R.T., Humphreys, P.K., A case based reasoning approach to make or buy decision. Integrated Manufacturing Systems 11 (5), Mohamed, A., Wanyama, T., Ruhe, G., Eberlein, A., Far, B., COTS evaluation supported by knowledge bases. Springer-Verlag, LSO 2004, LNCS 3096, pp Mollaghasemi, M., Pet-Edwards, J., Technical briefing: making multiple objective decisions. IEEE Computer Society Press, Los Alamitos, CA. Ncube, C., Dean, J.C., The limitations of current decision making techniques in the procurement of COTS software components. In: Proceedings of the First International Conference on COTS-based Software System, Orlando, February 2002, pp Nikoukaran, J., Hlupic, V., Paul, R.J., Criteria for simulation software evaluation. In: Proceedings of 1998 Winter Simulation Conference. Ngai, E.W.T., Chan, E.W.C., Evaluation of knowledge management tools using AHP. Expert Systems with Applications, Ochs, M., Pfahl, D., Chrobok-Diening, G., Nothhelfer-Kolb, B., A method for efficient measurement-based COTS Assessment and Selection method description and evaluation results. In: Proceedings of the 7th International Symposium on Software Metrics, pp Oh, K., Lee, N., Rhew, S., A selection process of COTS components based on the quality of the software in a special attention to internet. In: HIS 2003, LNCS 2713, pp Ossadnik, W., Lange, O., AHP-based evaluation of AHP-software. European Journal of Operational Research 118, Patel, N., Hlupic, V., A methodology for the selection of knowledge management (KM) tools. In: 24th International Conference on IT Interfaces, Cavtat, Croatia. Perini, A., Ricca, F., Susi, A., Tool-supported requirements prioritization: comparing the AHP and CBRank methods. Information and Software Technology 51, Pal, K., Campbell, J., An application of rule based and case based reasoning within a single legal knowledge-based system. The Database for Advances in Information Systems 28 (4), Plessis, A.L., A method for case tool evaluation. Information and Management 25, Reed, C., A minimum set of criteria for selecting a turn-key geographic information system. Computers, Environment and Urban Systems 7, Reimann, B.C., Waren, A.D., User-oriented criteria for the selection of DSS software. Communications of the ACM 28 (2). Ricci, F., Nguyen, Q.N., Acquiring and revising preferences in a critique-based mobile recommender system. IEEE Intelligent Systems 22, Ricci, F., Arslan, B., Mirzadeh, N., Venturini, A., A case-based travel advisory system. In: Proceedings of the 6th European Conference on Case-Based Reasoning. Springer-Verlag, pp Shtub, A., Spiegler, I., Kapaliuk, A., Using DSS methods in selecting operations management software. Computer Integrated Manufacturing Systems 1 (4), Sonar, R.M., An enterprise intelligent system development and solution framework. International Journal of Applied Science, Engineering and Technology 4 (1). Sonar, R.M., A web-based hybrid intelligent system framework. In: Proceedings of Sixth IASTED International Conference on Intelligent Systems and Control, ISC 2004, Honolulu, USA. Stahl, A., Combining case-based and similarity based product recommendation. In: ECCBR 2006, LNAI 4106, pp Stylianou, A.C., Madey, G., Smith, R.D., Selection criteria for expert system shells: a socio-technical framework. Communications of the ACM 32 (10). Tan, K.H., Lim, C.P., Platts, K., Koay, H.K., An intelligent decision support system for manufacturing technology investments. International Journal of Production Economics, Omega 27, Tewoldeberhan, T.W., Verbraeck, A., Valentin, E., Bardonnet, G., An evaluation and selection methodology for discrete-event simulation software. In: Proceedings of 2002 Winter Simulation Conference, pp Toshtzar, M., Multi-criteria decision making approach to computer software evaluation: application of the Analytical Hierarchical Process. Mathematical and Computer Modelling 11, Vlahavas, I., Stamelos, I., Refanidis, I., Tsoukias, A., ESSE: an expert system for software evaluation. Knowledge Based Systems 12, Yoon, K., Hwang, C., Multiple Attribute Decision-Making: An Introduction. Sage Publisher.