Draft. Evaluation of project and portfolio Management Information Systems with the use of a hybrid IFS-TOPSIS method

Transcription

1 Intellgent Decson Technologes 7 (2013) DOI /IDT IOS Press Evaluaton of project and portfolo Management Informaton Systems wth the use of a hybrd IFS-TOPSIS method Vassls C. Geroganns a,, Panos Ftsls a and Achlles D. Kameas b a Project Management Department, Technologcal Educaton Insttute of Larssa, Larssa, Hellas b Hellenc Open Unversty, Patras, Hellas Abstract. Contemporary Project and Portfolo Management Informaton Systems (PPMIS) have embarked from sngle-user, sngle-project management systems to web-based, collaboratve, mult-project, mult-operatonal nformaton systems whch offer organzaton-wde management support. The varety of offered functonaltes, along wth the varaton among each organzaton needs and the plethora of PPMIS avalable n the market, make the selecton of an approprate PPMIS a complcate, mult-crtera decson problem. The problem complexty s further augmented snce the mult stakeholders nvolved n the evaluaton/selecton process cannot often rate precsely ther preferences and the performances of canddate PPMIS on them. To meet these challenges, ths paper presents a PPMIS selecton/evaluaton approach that apples a hybrd group decson makng method based on TOPSIS and Intutonstc Fuzzy Sets (IFS). The approach consders the vagueness of assessors judgments when evaluatng PPMIS and the uncertanty of users when they judge ther needs. The approach s demonstrated through a case study amng to support the Hellenc Open Unversty to select a sutable PPMIS. Keywords: Project and portfolo management nformaton systems, mult-crtera decson makng, group decson makng, technque for order preference by smlarty to deal soluton, ntutonstc fuzzy sets 1. Introducton The adopton of an approprate Project and Portfolo Management Informaton System (PPMIS) offers a lot of benefts for an organzaton that undertakes projects, project programs and project portfolos to mplement busness process changes and develop new products or servces. Research studes [20] present that ncreasng organzatonal requrements for the management of the entre lfe-cycle of complex projects, programs and portfolos motvate the further explotaton of powerful PPMIS from organzatons of any type and sze. PPMIS have embarked Correspondng author: Vassls C. Geroganns, Project Management Department, Technologcal Educaton Insttute of Larssa, Larssa 41110, Hellas. Tel.: ; Fax: ; E-mal: gerogan@telar.gr. from stand-alone, sngle-user, sngle-project management systems to mult-user, mult-functonal, collaboratve, web-based and enterprse-wde software tools whch offer ntegrated project, program and portfolo management solutons, not lmted to scope, budget and tme management/control. Contemporary PPMIS can support, through a range of functonaltes, most processes n all knowledge areas of the Project Management Body of Knowledge [25], by coverng an expansve vew of the ntegraton management knowledge area that ncludes algnment and control of multproject programs and portfolos. The market of PP- MIS s rapdly growng and ncludes many commercal and open source software tools offerng a number of functonaltes such as tme, resource and cost management, reportng features and support for change, rsk, communcaton, contract and stakeholder management. Interested readers are referred to [22] where detaled nformaton s gven for 24 commercal lead-

2 92 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems ng PPMIS. In ths report, each presented PPMIS s evaluated upon approxmately 270 functonal and nonfunctonal features. Apart from commercal systems, for organzatons whch do not requre the complete range of functonaltes of a commercal tool or nterested n reducng the total cost of software ownershp, there s avalable a varety of open source PPMIS or Software as a Servce (SaaS) products. Ths varety of offered functonaltes, along wth the varaton among each organzaton needs and the large number of powerful PPMIS n the market, make ther evaluaton and selecton a complcate mult-crtera decson problem. The problem s often approached n practce by ad hoc procedures based only on personal preferences of users/evaluators or any marketng nformaton avalable [27]. Such an approach may lead to a fnal selecton that does not reflect adequately the organzaton needs or, even worse, to an unsutable PPMIS. Therefore, a systematc technque from the mult-crtera decson makng (MCDM) doman can be useful to support the PPMIS evaluaton/selecton process. The man objectve of ths paper s to present a possble soluton for ths mult-crtera decson problem. The paper presents a PPMIS evaluaton/selecton approach that apples a hybrd group decson makng method based on the Technque for Order Preference by Smlarty to Ideal Soluton (TOPSIS) [11] and Intutonstc Fuzzy Sets (IFS) [4]. The am of the approach s to consder the vagueness of assessors judgments when evaluatng PPMIS and the uncertanty of users when they judge ther needs. The approach s demonstrated through a case study amng to support the Hellenc Open Unversty to select a sutable PPMIS. The outlne of the paper s structured as follows. Secton 2 brefly revews the relevant lterature n the feld of MCDM technques for the evaluaton of software products. In Secton 3, we dscuss the aspects of the PPMIS evaluaton problem and we justfy how, n our case study, the PPMIS selecton crtera were determned. In Secton 4, we present an overvew of the characterstcs of the presented approach and justfy the selecton of IFS. In Secton 5, the basc concepts of IFS are brefly dscussed. Secton 6 presents the steps of the approach and Secton 7 presents conclusons and future work. 2. MCDM methods for evaluatng software packages Although there s no a generc MCDM approach for selectng a software package of any type, avalable lterature revews n software products evaluaton [19] suggest that users and evaluators can receve a lot of benefts f they decde to adopt a MCDM method. Revew surveys [18,19,29] reveal that the Analytc Herarchy Process method (AHP) and ts varatons/extensons have been wdely and successfully used n evaluatng several types of software packages (e.g., MRP systems, ERP systems, smulaton software, CAD systems and knowledge management systems). Ths extensve applcaton of AHP s due to the method advantages, snce t supports the herarchcal decomposton of a decson problem, allows decson makng to be held by a group of stakeholders as well as t handles both qualtatve and quanttatve selecton crtera. Although AHP presents wde applcablty n evaluatng varous types of software products, lttle work has been done n the feld of evaluatng PPMIS. For example, n [2] the authors admt that ther work s rather ndcatve wth man objectve to expose a representatve case for llustratng the PPMIS selecton process and not to create a defntve set of crtera that should be taken nto account n practce. Ths lack of applcablty of AHP n the PPMIS selecton problem doman can be attrbuted to the fact that, despte ts advantages, the method man lmtaton s the large number of parwse comparsons requred. The tme needed for comparsons ncreases geometrcally wth the ncrease of crtera and alternatves nvolved, makng AHP applcaton practcally prohbtve for complcate decsons, such as the selecton of a PPMIS. As a response to ths problem, n the recent past, we presented an approach for evaluatng alternatve PP- MIS that combnes group-based AHP wth a smple scorng model [15]. Accordng to ths approach, PP- MIS evaluators (decson makers) use a scorng model to evaluate the performances of canddate systems wth respect to an extensve lst of functonal-orented crtera (organzed nto crtera clusters), whle PPMIS users follow the AHP method to determne the overall weghts of the crtera clusters based on the needs of ther organzaton. Ths group-ahp scorng model, although practcal and easy to use, does not consder the vagueness or even the unawareness of users, when they evaluate ther preferences from a PPMIS by ratng ther requrements. Also the approach does not take nto account the uncertanty of evaluators, when they judge the performance ratngs of alternatve PPMIS on the selected crtera, expressed as user requrements. In case of evaluatng a PPMIS, these uncertantes are more evdent when the software product does not offer a certan functonalty by default (n the product

3 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems 93 standard verson) and the desred functonalty can be fulflled at a certan degree through confguraton, customzaton, use of workarounds (other functonaltes that act as substtutes) or use of nterfaces to other software products. Treatng wth these ambgutes n the PPMIS evaluaton and consderaton of ncomplete avalable nformaton expose the need to adopt a fuzzy-based decson makng approach [9]. Fuzzy-based methods provde the ntutve advantage to utlze, nstead of crsp values, lngustc terms to evaluate performance of the alternatves and crtera weghts. A lngustc term (.e., a varable whose value s a natural language phrase) can be partcularly useful to express qualtatve assessments. A fuzzy-based approach can be even more benefcal when t s combned wth other decson makng technques. For example, fuzzy AHP [10] s proposed to handle the nherent mprecson n the parwse comparson process, whle fuzzy TOPSIS (Technquefor Order PreferencebySmlarty to Ideal Soluton) [11] can be used to jontly consder both postve (beneft/functonal orented) and negatve (cost/effort orented) selecton crtera. Fuzzy-based MCDM technques have been used to select varous types of software products (see, for example [9,13,21]), but n the relevant lterature there s lack of a structured fuzzybased approach for the selecton of PPMIS under uncertan knowledge. Ths paper presents such a fuzzy-based approach that n comparson wth other MCDM approaches for software product selecton [9,13,21] manly dffers n three aspects: ) the approach nvolves both decson makers and PPMIS users n the decson makng process and aggregates ther weghted opnons (through fuzzy weghted averagng operators) to support agreement upon the fnal selecton, ) the approach handles the degree of ndetermnacy that characterzes both decson makers and users n ther evaluatons, ) both postve (beneft) and negatve (cost) crtera are consdered n the evaluaton. These aspects are supported by the approach underlng method that s a hybrd group decson makng method based on TOPSIS and Intutonstc Fuzzy Sets. 3. PPMIS adopton, evaluaton and selecton crtera Emprcal studes [26] demonstrate that a number of project managers from the busness communty ndcate a strong mpact of PPMIS usage upon successful mplementaton of ther busness projects, whle others do not. These fndngs ndcate that unsatsfed project managers are depended upon a PPMIS that produces nformaton of low qualty. Hence project managers use the system less and consequently get less support n ther management tasks. It s mportant, therefore, for an enterprse to select a proper PP- MIS that covers techncal, manageral and organsatonal needs. The mportance of project management technques and tools s also ganng recognton n academc nsttutons and Hgher Educaton organzatons as a valuable tool, especally for supportng unversty Informaton Technology (IT) projects. The results of a survey n IT departments of US unverstes show that project plannng, montorng and status/budget reportng are of crucal mportance for the majorty of unversty projects [34]. The PPMIS selecton process can be supported by referencng to avalable market surveys [27]. In the past, for example, the Project Management Insttute has publshed an extensve survey [24] that compared more than 200 products by consderng classc project management dmensons lke schedulng, cost, rsk, resource and communcaton management. These comparsons, however, focus rather on factors whch represent vendors perspectves and any such assessment should be utlzed wth care by consderng specfc project management needs wthn the context of ndvdual organzatons. Furthermore, PPMIS of today offer support for the entre project lfe-cycle, ncludng portfolo plannng and montorng and thus, a PPMIS evaluaton process based only on classc sngle-project management functonaltes s very lmted. Support n settng up a PPMIS system can be also ganed by consderng the users perceptons and satsfacton from a PPMIS usage. A representatve example s the one presented n [3]. Ths work surveyed 497 PPMIS users and the fnal result was a general ndex for measurng the effectveness of PPMIS accordng to four, user-orented, dmensons (.e., nformaton qualty, system functonalty, ease of use, performance mpact). However, respectve PPMIS users, when evaluate a PPMIS system, often express ther perceved satsfacton and not ther knowledge on potental benefts that can be obtaned from a PPMIS. Detaled assstance n evaluatng PPMIS s provded by evaluaton frameworks whch propose to consder an extensve lst of system characterstcs. These characterstcs can be ether functonal or process-orented selecton crtera. NASA, for example, n the past has convened a workng group to evaluate alternatve PP-

4 94 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems Fg. 1. M-Model archtecture source: [6]. (Colours are vsble n the onlne verson of the artcle; MIS for NASA s departments, upon a number of functonal requrements. In the group s report [16] thrteen clusters of functonal requrements are dentfed, namely: 1) open database connectvty and archtecture, 2) workgroup capabltes, 3) networkng capabltes, 4) ease of use, 5) project schedulng methodology, 6) project task/feld features, 7) baselnng and trackng project progress, 8) resource features, 9) calendar features, 10) cost management features, 11) rsk management features, 12) project reportng, and 13) management reportng. Each cluster further ncludes a set of functonal features and, n total, more than one hundred functonal crtera are dentfed to be evaluated. Ths vast number of crtera prevents decson makers from utlzng a typcal herarchcal MCDM approach lke AHP. As far as process orented evaluaton s concerned, evaluators may use as reference the set of processorented crtera offered by a conceptual software archtecture for PPMIS, lke, for example, s the M- Model (Fg. 1) [1]. An abstract software archtecture may be used to handle the selecton problem from a busness process reengneerng perspectve, snce t embraces all tasks performed durng a project/program lfe-cycle (ntaton, plannng, executon and termnaton). The M-Model specfes the project phases/tasks supported by PPMIS whch are mapped nto dfferent management levels (project, program and portfolo management). The model was used n [22] to evaluate commercal PPMIS accordng to the project phases/tasks supported by a PPMIS and the correspondng requred functonaltes (Table 1). Each PP- MIS was evaluated accordng to the extent that t offers the requred functonalty and the overall support for the correspondng project phase/task was specfed wth a 4-stars score. Yellow and grey stars at each score ndcate (see Fg. 1) f the correspondng support s offered as standard functonalty (yellow stars) or t can be provded by customzng the PPMIS (grey stars). The authors admt n ther evaluaton report that the stars (.e., the performance ratng) assgned at each PPMIS upon each crteron (.e., the level of provded support for the correspondng project phase/task) are obtaned only by countng the number of functonaltes offered by the standard system verson or through applyng smple customsatons (wthout ex-

5 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems 95 Table 1 Evaluaton crtera source: [22] Phase/task Requred functonalty 1. Idea generaton/lead management (IGLM) Creatvty technques, dea/project classfcaton, lead management (Mgmt.), project status/project process mgmt 2. Idea evaluaton (IE) Estmaton of effort, resource needs specfcaton, rsk estmaton, proftablty analyss, project budgetng, offer mgmt 3. Portfolo plannng (PP1) Organzatonal budgetng, project assessment, project portfolo optmzaton, project portfolo confguraton 4. Program plannng (PP2) Project templates, resource master data, resource assgnment workflow, resource allocaton 5. Project plannng (PP3) Work breakdown structure plannng, scope/product plannng, network plannng, schedulng, resource levelng, rsk plannng, cost plannng 6. Project controllng (PC1) Change request Mgmt., (travel) expense Mgmt., tmesheet, cost controllng, meetng support 7. Program controllng (PC2) Status reportng, devaton/earned value analyss, qualty controllng, versonng, mlestone controllng 8. Portfolo controllng (PC3) Performance measurement, dashboard, organzatonal budget controllng 9. Program termnaton (PT1) Knowledge portal, competence database/yellow pages, project archvng, searchng 10. Project termnaton (PT2) Invocng, document Mgmt., suppler and clam Mgmt. 11. Admnstraton/confguraton (AC) Workflow Mgmt., access control, report development, form development, user-defned data structures, MS offce project nterface, applcaton programmng nterface, offlne usage tensve code development). Thus, any workarounds, nterfaces and/or use of programmng (n case of open source PPMIS) are not taken nto account n ths evaluaton report. In case of selectng a proper PPMIS for a specfc organzaton, the consderaton of these parameters or lack of knowledge upon them wll certanly affect the uncertanty of the fnal performance ratng for each canddate PPMIS. In Secton 6 of the paper, we show how these 11 project phases/tasks (Table 1) were ncluded n a lst of selecton crtera for evaluatng alternatve PPMIS for the case organzaton. Ths decson supported users (members of the case organzaton) to rate the mportance of ther requrements by consderng the processes supported by PPMIS, wthout need for knowng techncal capabltes and functonaltes of each canddate system. Ths decson also helped decson makers (evaluators) to perform cross-checkng (.e., comparsons) of ther lngustc assessments on canddate PPMIS aganst the scores presented n [22]. In addton, the adopton of ntutonstc fuzzy numbers for the evaluaton of PPMIS helped decson makers to consder n the evaluaton not only the avalablty/unavalablty of a functonalty to support a crteron but also the degree that the unavalablty can be relaxed though admttng that there can be also other solutons, not offered by the out of the box PPMIS verson. 4. Overvew of the suggested PPMIS evaluaton/selecton approach The suggested approach for PPMIS nvolves both users and evaluators (decson makers) n the decson makng process and tres to explot the nterest/expertse of each one n order to strengthen the fnal evaluaton results. Ths s acheved by aggregatng all weghts of crtera (requrements) and all ratngs of performance of the alternatve systems, as they are expressed, by ndvdual stakeholders, n lngustc terms. The approach s based on Intutonstc Fuzzy Sets (IFS), an extenson of fuzzy sets proposed by Atanassov [4] that has been successfully used n many decson makng problems, such as medcal dagnoss [14], web servces selecton [33] and suppler selecton [7, 12,23]. An IFS ncludes the membershp and the nonmembershp functon of an element to a set as well as a thrd functon that s called the hestaton degree. Ths thrd functon s useful to express lack of knowledge and hestancy concernng membershp and nonmembershp of an element to a set. Expresson of hestaton s partcularly helpful for both decson makers and PPMIS users when they select a software product for an organzaton such as, n our case, a PPMIS. On one hand, decson makers often cannot have a full knowledge upon all functonaltes ncluded n the newest verson of each canddate system. Thus, they base ther ratngs only on ex-

6 96 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems perence from usng prevous system versons as well by referencng to system assessments whch can be found n products survey reports [22]. Furthermore, a negatve PPMIS performance ratng that s assgned when the system does not provde standard support for a requred functonally (.e., the functonalty s not avalable at the standard verson of the system) can be even more hestant, snce the software may offer the functonalty through confguraton, customzaton, use of workarounds (other functonaltes that act as substtutes) or nterfaces to other avalable software products. On the other hand, PPMIS users are often unfamlar wth how a system can support project management processes and tasks and, therefore, cannot precsely express whch tasks requre more to be supported by a PPMIS. It should be noted here that the presented approach manly utlzes a hybrd method presented n [7] whch combnes IFS wth TOPSIS for supportng suppler selecton problems. The advantage of ths combnaton n case of PPMIS evaluaton s that we can dstngush between beneft crtera (e.g., functonaltes/tasks supported by the PPMIS) and cost crtera (e.g., effort for system customsaton and prce for ownershp). The PPMIS that s closest to the postve deal soluton and most far from the negatve deal soluton could be probably the most approprate PPMIS to cover the organzaton needs. The approach not only valdates the orgnal method n a new applcaton feld that s the evaluaton of PPMIS (where other MCDM approaches are rather lmted n the lterature), but also consders a more extensve lst of beneftand cost orented crtera, sutable for PPMIS selecton. In addton, fnal results are verfed by applyng senstvty analyss. 5. Intutonstc fuzzy sets: Basc concepts Before proceedng to descrbe how the PPMIS selecton problem was tackled, we brefly ntroduce some necessary ntroductory concepts of IFS. An IFS A n a fnte set X can be defned as [4]: A = {< x,μ A (x),v A (x) > x X} (1) where μ A : X [0, 1], v A : X [0, 1], and0 μ A (x)+v A (x) 1 x X. μ A (x) and v A (x) denote respectvely the degree of membershp and non-membershp of x to A. For each IFS A n X, π A (x) =1 μ A (x) v A (x) s called the hestaton degree of whether x belongs to A. Ifthe hestaton degree s small then knowledge whether x Table 2 Lngustc terms for the mportance of stakeholders and crtera Level of stakeholder expertse (1) Importance of selecton crtera (2) IFN(3) Master Very mportant (VI) [0.90,0.10] Expert Important (I) [0.75,0.20] Profcent Medum (M) [0.50,0.45] Practtoner Unmportant (U) [0.25,0.70] Begnner Very unmportant (VU) [0.10,0.90] belongs to A s more certan, whle f t s large then knowledge on that s more uncertan. Thus, an ordnary fuzzy set can be wrtten as: {< x,μ A (x), 1 μ A (x) > x X} (2) In the evaluaton approach we wll use lngustc terms [17] to express: ) the mportance of decson stakeholders (users/decson makers), ) judgements of decson makers on the performance of each PPMIS and ) perceptons of users on the mportance of each selecton crteron. These lngustc terms can be transformed nto ntutonstc fuzzy numbers (IFNs) n the form of [μ(x), v(x)]. For example, an IFN [0.50, 0.45] represents membershp μ = 0.5, non-membershp v = 0.45 and hestaton degree π =0.05. In the approach, we wll also use addton and multplcaton operators for IFNs. Let a1 =(μ a1,v a1 ) and a2 =(μ a2,v a2 ) be two IFNs. Then these operators can be defned as follows [4,30,31]: a1 a2=(μ a1 + μ a2 μ a1 μ a2,v a1 v a2 ) a1 a2=(μ a1 μ a2,v a1 +v a2 v a1 v a2 ) (3) λ a1=(1 (1 μ a1 ) λ,v λ a1),λ>0 6. Evaluaton of PPMIS wth ntutonstc fuzzy sets and TOPSIS In ths secton we descrbe how an ntutonstc fuzzy MCDM approach was appled wth the overall goal to select the most approprate PPMIS system to cover needs of the Hellenc Open Unversty (HOU) ( n facltatng, supportng and provdng project management for unversty-ndustry collaboraton n research and development (R&D). HOU s a unversty that undertakes varous types of natonal and nternatonal R&D projects and programs, partcularly n the feld of contnuous adult educaton. The unversty does not mantan an ntegrated project/portfolo management nfrastructure. In order to ncrease project management maturty, effectveness and productvty, the management of HOU has decded to nvestgate the

7 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems 97 Table 3 Lngustc terms for ratng the performance of PPMIS Level of performance/support IFN Degree of hestaton (π) Fnal IFN Extremely hgh (EH) [1.00,0.00] 0 [1.00,0.00] Very very hgh (VVH) [0.90,0.10 π] 0 [0.90,0.10] Very hgh (VH) [0.80,0.20 π] 0.1 [0.80,0.10] Hgh (H) [0.70,0.30 π] 0.1 [0.70,0.20] Medum hgh (MH) [0.60,0.40 π] 0.1 [0.60,0.30] Medum (M) [0.50,0.50 π] 0.1 [0.50,0.40] Medum low (ML) [0.40,0.60 π] 0.1 [0.40,0.50] Low (L) [0.30,0.70 π] 0.1 [0.30,0.60] Very low (VL) [0.20,0.80 π] 0 [0.20,0.80] Very very low (VVL) [0.10,0.90 π] 0 [0.10,0.90] Fg. 2. Steps of the PPMIS evaluaton approach. (Colours are vsble n the onlne verson of the artcle; adopton of a collaboratve PPMIS. The Department of Project Management (DPM) (dde.telar.gr) at the Technologcal Educaton Insttute of Larssa n Greece was apponted to act as an experenced consultant and ad ths decson makng process. Three experts D 1, D 2 and D 3 (decson makers/ evaluators) from DPM, wth an average of seven years teachng/professonal experence n usng dfferent PP- MIS, were nvolved n ths process, amng to dentfy HOU requrements from a PPMIS and to select an approprate system that wll cover these requrements. Three project offcers/managers U 1, U 2 and U 3 (users) from the HOU ste were also nvolved n the decson makng. These persons have hgh expertse n contract management, mult-project coordnaton and plannng of R&D projects and portfolos, but they present low experence n systematcally usng PPMIS. The applcaton of the approach for selectng an approprate PPMIS for the case organzaton (HOU) has been conducted n eght steps (Fg. 2) presented as follows. Step 1: Determne the weght of mportance of decson makers and users In ths frst step, the expertse of both decson makers and users was analysed by specfyng correspondng weghts. In a jont meetng, the three decson makers D 1, D 2, D 3 agreed to qualfy ther experence n usng PPMIS as Master, Profcent and Expert, respectvely. The three users U 1, U 2, U 3 also agreed that ther level of expertse n managng large projects can be characterzed as Master, Profcent and Expert, respectvely. These lngustc terms were assgned to IFNs by usng the relatonshps presented n Table 2 between values n column 1 and values n column 3. If there are l stakeholders n the decson process, each one wth a level of expertse rated equal to the IFN [μ k, v k, π k ], the weght of mportance of k stakeholder can be calculated as [7]: ( ( )) μk μ k + π k λ k = l ( μ k + π k ( μ k + v k μk μ k + v k )) (4) where λ k [0, 1] and l λ k =1. By applyng Eq. (4) the weghts of decson makers were calculated as follows: λ D1 =0.406, λ D2 = 0.238, λ D3 = Snce users were assgned to the same lngustc values, ther weghts were respectvely the same: λ U1 =0.406, λ U2 =0.238, λ U3 = It should be noted here that the heurstc of Eq. (4) for

8 98 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems Table 4 The ratngs of the alternatve PPMIS Crtera Decson makers PPMIS A 1 A 2 A 3 A 4 A 5 IGLM D 1 VH VH H MH H D 2 H VH MH H H D 3 H H H H MH IE D 1 H M VH M M D 2 MH M H H H D 3 M MH H MH H PP1 D 1 MH H VVH VH VH D 2 MH MH VH MH VH D 3 MH MH H H VH PP2 D 1 MH MH VH VH VH D 2 MH H VH MH VH D 3 H M H MH H PP3 D 1 VH H VH VH VH D 2 H H MH H VH D 3 VH VH H MH MH PC1 D 1 H VH VH VH H D 2 MH H H H H D 3 H H H MH MH PC2 D 1 H MH VH H VH D 2 MH M H H VH D 3 H MH H MH H PC3 D 1 H VH MH H VH D 2 MH H M H VH D 3 H MH H VH M PT1 D 1 H H VH VH H D 2 MH H VH H MH D 3 H MH H MH M PT2 D 1 H H VH H VH D 2 H H VVH MH VH D 3 H MH H MH H AC D 1 MH H H H M D 2 M M MH MH MH D 3 H MH H MH M PO D 1 MH MH H H VH D 2 M M MH MH H D 3 MH MH H MH H CC D 1 M MH MH H VH D 2 M M MH MH VH D 3 H MH M MH H calculatng weghts has been also adopted n other selecton methods (see, for example [7,32,35]). Step 2: Determne the level of support provded by each alternatve PPMIS Though there s a large number of avalable PPMIS, decson makers were quered to express ther general opnon on ten commercal PPMIS whch n market survey results [27] are charactersed as leaders and challengers n ths segment of enterprse software market. Fve from these systems were excluded for two reasons. Frst, snce they do not have presence n the natonal market and, second, because decson makers were persuaded that ther usage was napproprate for the specfc case, manly due to lack of techncal support and non-avalablty of tranng servces. Ths frst-level screenng resulted n a lst of fve powerful, wdespread PPMIS wth strong presence (.e., techncal/tranng support) n the natonal market. For confdentalty reasons and amng at avodng the commercal promoton of any software package, we wll refer to these PPMIS as A 1, A 2, A 3, A 4 and A 5. In order to evaluate the canddate PPMIS n a manageable and relable way, decson makers (evaluators) rated the performance of each system wth respect to the crtera prevously dentfed. Each decson maker was asked to carefully rate the support provded by each system on each of the 11 crtera (project phases/tasks) presented n Table 1. In addton to these 11 postve (beneft orented) crtera, two negatve (cost orented crtera) were decded to be ncluded n the lst. These are the total prce for purchasng/ownershp (PO) and the effort requred to customse/confgure the PPMIS (CC). Thus, 13 crtera n total were adopted. All decson makers provded a short wrtten justfcaton for every ratng they gave n lngustc terms. For ther ratngs decson makers used the lngustc terms presented n Table 3. For the constructon of Table 3, the so-called Postve- Confdence Approach [33] was adopted, accordng to whch the degree of support offered by an evaluated system to a certan crteron s made frm (.e, the membershp value), and the assocated hestaton degree s subtracted from the degree that the system does not support the crteron (.e, the non-membershp value). Decson makers expressed n a jont meetng that they are rather confdent n ther judgements and they decded hestaton degrees equal to 0 and 0.1 for strong judgments (.e., EH, VVH, VL, VVL) and medum judgments (VH, H, MH, M, ML, L), respectvely. Decson makers justfed ths agreement upon the hestaton degrees by commentng that: ) they have experence n utlzng these 5 canddate PPMIS, and thus they feel qute determnant n ther judgments and ) the canddate systems are commercal tools (and not open source products) and the level of functonalty that can be easly mplemented (by confguraton) to acheve a not- supported functonalty s low. To check the valdty of the ratngs, decson makers were also asked to cross-check ther marks, accordng to the correspondng 4-stars scores, as they are lsted for each tool n [22]. All ratngs fnally gven by the three decson makers to the fve PPMIS alternatves are presented n Table 4.

9 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems 99 Table 5 Aggregated ntutonstc fuzzy decson matrx A 1 A 2 A 2 A 4 A 5 IGLM IE PP PP PP PC PC PC PT PT AC PO CC Based on these ratngs and the weghts of decson makers, the aggregated ntutonstc fuzzy decson matrx (AIFDM) was calculated by applyng the ntutonstc fuzzy weghted averagng (IFWA) operator [31]. The basc steps of the IFWA operator are that t frst weghts all gven IFNs by a normalzed weght vector, and then aggregates these weghted IFNs by addton. Each result derved by usng the IFWA operator s an IFN. If A = {A 1,A 2,...,A m } s the set of alternatves and X = {X 1,X 2,...,X n } s the set of crtera, then AIFDM R s an m n matrx wth elements IFNs n the form of r j = [μ A (x j ),v A (x j ),π A (x j )], where = 1, 2,...,m and j =1, 2,...,n. By consderng weghts λ k (k =1, 2,...,l) of l decson makers, the elements r j of the AIFDM can be calculated usng IFWA as follows: r j = IFWA λ (r (1) j,r(2) j,...,r(l) j ) = λ 1 r (1) j λ 2 r (2) j λ 3 r (3) j... λ l r (l) j [ = 1 l (1 μ (k) j )λ k l, (v (k) j )λ k (5), ] l (1 μ (k) l j )λ k (v (k) j )λ k The AIFDM for the case problem s shown n Table 5. The matrx IFNs were calculated by substtutng n Eq. (5) the weghts of the three (l =3) decson makers (λ D1 =0.406, λ D2 =0.238, λ D3 =0.356) and the IFNs (μ (k) j,v(k) j,π(k) j ) produced by usng the relatonshps of Table 3 (.e., these IFNs correspond to ratngs gven by the k decson maker on each system A ( =1, 2,...,5) wth respect to each crteron j (j =1, 2,...,13). For example, n Table 5, the IFN [0.769, 0.128, 0.103], shown n bold, s the aggregated score of PP- MIS A 2 on crteron IGLM (Idea Generaton/Lead Mgmt.), whle the IFN [0.600, 0.300, 0.100], also shown n bold, s the aggregated score of PPMIS A 1 on crteron PP1 (Portfolo Plannng). Step 3: Determne the weghts of the selecton crtera To analyse users requrements from a PPMIS we dssemnated to the three users/members of HOU a structured questonnare, askng from them to evaluate the 13 selecton crtera and express ther perceptons on the relatve mportance of each one crteron wth respect to the overall performance and benefts provded from a canddate PPMIS. Each of the 3 users was requested to answer 13 questons by denotng a grade for the mportance of each crteron n a lngustc term, as t s shown n column 2 of Table 2. Opnons of users U 1, U 2 and U 3 on the mportance of the crtera are presented n Table 6. These preferences are assgned to correspondng IFNs by usng the relatonshps between values n column 2 and values n column 3 of Table 2. The IFWA operator was also used to calculate the weghts of crtera by aggregatng the opnons of the users. Let w (k) j = (μ (k) j,v (k) j,π (k) j ) be the IFN assgned to crteron j (j =1, 2,...,n)bythek user (k =1, 2,...,l). Then the weght of j can be calculated as follows:

10 100 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems Table 6 Importance values of the crtera Crtera Users U 1 U 2 U 3 IGLM VI I M IE M VI I PP1 M VI VI PP2 VI VI VI PP3 I VI VI PC1 M VI VI PC2 M VI I PC3 M M VI PT1 I VI VI PT2 VI M I AC VI I I PO VI VI M CC I M VI Table 7 Weghts of the crtera Crtera Weghts μ v π IGLM IE PP PP PP PC PC PC PT PT AC PO CC w j =IFWA λ (w (1) j,w (2) j,...,w (l) j ) =λ 1 w (1) j λ 2 w (2) j λ 3 w (3) j... λ l w (l) j [ = 1 l (1 μ (k) j ) λ k l, (v (k) j ) λ k, ] l (1 μ (k) j ) λ k l (v (k) j ) λ k (6) Thus, a vector of crtera weghts s obtaned W = [w 1,w 2,...,w j ], where each weght w j s an IFN n the form [μ j, v j, π j ](j = 1, 2,...,n). In the case problem, substtutng n Eq. (6) the weghts of three users (λ U1 =0.406, λ U2 =0.238, λ U3 =0.356)and usng the IFNs whch correspond to lngustc values of Table 6 yelded the crtera weghts shown n Table 7. Step 4: Compose the aggregated weghted ntutonstc fuzzy decson matrx In ths step, the aggregated weghted ntutonstc fuzzy decson (AWIFDM) matrx R s composed by consderng the aggregated ntutonstc fuzzy decson matrx (.e., table R produced n step 2) and the Table 8 Aggregated weghted ntutonstc fuzzy decson matrx A 1 A 2 A 3 A 4 A 5 IGLM IE PP PP PP PC PC PC PT PT AC PO CC vector of the crtera weghts (.e., table W produced n step 3). Step 4 s necessary to synthesze the ratngs of both decson makers and users. In partcular, elements of the AWIFDM can be calculated by usng the multplcaton operator of IFS as follows: R W = {< x,μ A (x) μ W (x),v A (x) +v W (x) v A (x) v W (x) > x X} (7) R s an m n matrx composed wth IFNs n the form of r j = [μ AW (x j ),v AW (x j ),π AW (x j )], where: μ AW (x j ), v AW (x j ) are values derved by applyng Eq. (7). The hestaton degree can be computed each tme by subtractng the sum of these two values

11 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems 101 Table 9 Separaton measures and relatve closeness coeffcent of each PPMIS PPMIS S (1) S (2) C (3) A A A A A (μ AW (x j ), v AW (x j )) from 1: π AW (x)=1 v A (x) v W (x) μ A (x) μ W (x)+v A (x) v W (x) (8) In the case problem, substtutng n Eq. (7) the IFNs of Table 5 (table R) and IFNs of Table 7 (table W ) yelded the IFNs of the AWIFDM (table R ) presented n Table 8. For example, n Table 8, the IFN [0.599, 0.304, 0.097], shown n bold, s the aggregated weghted score of PPMIS A 2 on crteron IGLM (Idea Generaton/Lead Mgmt.), whle the IFN [0.485, 0.429, 0.086], also shown n bold, s the aggregated weghted score of PPMIS A 1 on crteron PP1 (Portfolo Plannng). Step 5: Compute the ntutonstc fuzzy postve deal soluton and the ntutonstc fuzzy negatve deal soluton To apply the TOPSIS method the ntutonstc fuzzy postve deal soluton (IFPIS) A and the ntutonstc fuzzy negatve deal soluton (IFNIS) A have to be determned. Both solutons are vectors of IFN elements and they are derved from the AWIFDM matrx as follows. Let B and C be the sets of beneftandcost crtera, respectvely. Then A and A are equal to: and where A =(μ A W (x j ),v A W (x j )) A =(μ A W (x j ),v A W (x j )) μ A W (x j ) = ((max μ AW (x j ) j B), μ AW (x j ) j C)) (mn v A W (x j ) = ((mn (max μ A W (x j ) = ((mn (max v A W (x j ) = ((max (mn v AW (x j ) j B), v AW (x j ) j C)) μ AW (x j ) j B), μ AW (x j ) j C)) v AW (x j ) j B), v AW (x j ) j C)) (9) In the case problem, B = {IGLM, IE, PP1, PP2, PP3, PC1, PC2, PC3, PT1, PT2, AC} and C = {PO, CC}. To obtan IFPIS and IFNIS, Eq. (9) was appled on the IFNs of the AWIFDM decson matrx. The IFPIS and IFNIS were determned as follows: A =([0.599, 0.304, 0.097], [0.547, 0.351, 0.102], [0.667, 0.289, 0.044], [0.692, 0.215, 0.093], [0.667, 0.235, 0.099], [0.602, 0.307, 0.090], [0.564, 0.334, 0.102], [0.532, 0.378, 0.090], [0.657, 0.244, 0.099], [0.641, 0.288, 0.072], [0.562, 0.338, 0.100], [0.476, 0.437, 0.087], [0.446, 0.459, 0.095]) A =([0.517, 0.390, 0.094], [0.395, 0.512, 0.094], [0.485, 0.429, 0.086], [0.536, 0.372, 0.092], [0.614, 0.284, 0.102], [0.539, 0.373, 0.088], [0.424, 0.481, 0.095], [0.445, 0.468, 0.087], [0.525, 0.377, 0.097], [0.513, 0.391, 0.096], [0.435, 0.468, 0.097], [0.613, 0.296, 0.091], [0.605, 0.293, 0.101]) Step 6: Calculate the separaton between the alternatve PPMIS Next, the separaton measures S and S can be calculated for each canddate system A from the IF- PPIS and the IFNIS, respectvely. As a dstance measure, the normalzed Eucldean dstance was adopted, snce t has been proved to be a relable dstance measure that takes nto account not only membershp and non-membershp but also the hestaton part of IFNs [28]. For each alternatve system these two separaton values can be calculated as follows: n 1 S 2n [(μ AW (x j ) μ A W (x j )) 2 = j=1 +(v AW (x j ) v A W (x j )) 2 +(π AW (x j ) π A W (x j )) 2 ] (10) n 1 S 2n [(μ AW (x j ) μ A W (x j )) 2 = j=1 +(v AW (x j ) v A W (x j )) 2 +(π AW (x j ) π A W (x j )) 2 ] By utlzng these Eq. (10), the postve and negatve separaton measures for the fve alternatve PPMIS were calculated. These are shown n columns (1) and (2) of Table 9.

12 102 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems Table 10 Senstvty analyss results (based on crtera weghts) Exp. Crtera weghts Scores of PPMIS Rankng A 1 A 2 A 3 A 4 A 5 1 w 1 13 =[0.10, 0.90] A 3 >A 4 >A 1 >A 5 >A 2 2 w 1 13 =[0.25, 0.70] A 3 >A 4 >A 1 >A 5 >A 2 3 w 1 13 =[0.50, 0.45] A 3 >A 4 >A 1 >A 5 >A 2 4 w 1 13 =[0.75, 0.20] A 3 >A 4 >A 1 >A 5 >A 2 5 w 1 13 =[0.90, 0.10] A 3 >A 4 >A 1 >A 5 >A 2 6 w 1 =[0.90, 0.10], w 2 13 =[0.10, 0.90] A 2 >A 1 >A 3 >A 5 >A 4 7 w 2 =[0.90, 0.10], w 1,3 13 =[0.10, 0.90] A 3 >A 5 >A 1 >A 4 >A 2 8 w 3 =[0.90, 0.10], w 1 2,4 13 =[0.10, 0.90] A 3 >A 5 >A 4 >A 2 >A 1 9 w 4 =[0.90, 0.10], w 1 3,5 13 =[0.10, 0.90] A 3 >A 5 >A 4 >A 2 >A 1 10 w 5 =[0.90, 0.10], w 1 4,6 13 =[0.10, 0.90] A 1 >A 3 >A 5 >A 2 >A 4 11 w 6 =[0.90, 0.10], w 1 5,7 13 =[0.10, 0.90] A 3 >A 2 >A 4 >A 1 >A 5 12 w 7 =[0.90, 0.10], w 1 6,8 13 =[0.10, 0.90] A 3 >A 5 >A 1 >A 4 >A 2 13 w 8 =[0.90, 0.10], w 1 7,9 13 =[0.10, 0.90] A 4 >A 5 >A 2 >A 1 >A 3 14 w 9 =[0.90, 0.10], w 1 8,10 13 =[0.10, 0.90] A 3 >A 4 >A 1 >A 2 >A 5 15 w 10 =[0.90, 0.10], w 1 9,11 13 =[0.10, 0.90] A 3 >A 5 >A 1 >A 2 >A 4 16 w 11 =[0.90, 0.10], w 1 10,12 13 =[0.10, 0.90] A 3 >A 4 >A 2 >A 1 >A 5 17 w 12 =[0.90, 0.10], w 1 11,13 =[0.10, 0.90] A 1 >A 2 >A 4 >A 3 >A 5 18 w 13 =[0.90, 0.10], w 1 12 =[0.10, 0.90] A 3 >A 1 >A 2 >A 4 >A 5 Table 11 Senstvty analyss results (based on performance ratngs) Exp. Performance ratngs Scores of PPMIS Rankng A 1 A 2 A 3 A 4 A 5 1 VVH=[0.9,0.1,0] VH=[0.7,0.3,0] H=[0.5,0.5,0] A 3 >A 4 >A 5 >A 1 >A 2 MH=[0.3,0.7,0] M=[0.1,0.9,0] Postve-confdence scale 2 VVH=[0.9,0,0.1] VH=[0.7,0.2,0.1] A 3 >A 5 >A 4 >A 1 >A 2 H=[0.5,0.4,0.1] MH=[0.3,0.6,0.1] M=[0.1,0.8,0.1] 3 VVH=[0.9,0,0.1] VH=[0.7,0.1,0.2] A 3 >A 4 >A 5 >A 1 >A 2 H=[0.5,0.3,0.2] MH=[0.3,0.5,0.2] M=[0.1,0.7,0.2] 4 VVH=[0.9,0,0.1] VH=[0.7,0,0.3] A 3 >A 4 >A 5 >A 1 >A 2 H=[0.5,0.2,0.3] MH=[0.3,0.4,0.3] M=[0.1,0.6,0.3] 5 VVH=[0.9,0,0.1] VH=[0.7,0,0.3] A 3 >A 4 >A 1 >A 5 >A 2 H=[0.5,0.1,0.4] MH=[0.3,0.3,0.4] M=[0.1,0.5,0.4] 6 VVH=[0.9,0,0.1] VH=[0.7,0,0.3] A 3 >A 4 >A 1 >A 5 >A 2 H=[0.5,0,0.5] MH=[0.3,0.2,0.5] M=[0.1,0.4,0.5] 7 VVH=[0.9,0,0.1] VH=[0.7,0,0.3] A 3 >A 4 >A 5 >A 1 >A 2 H=[0.5,0,0.5] MH=[0.3,0.1,0.6] M=[0.1,0.3,0.6] Negatve-confdence scale 8 VVH=[0.8,0.1,0.1] VH=[0.6,0.3,0.1] A 3 >A 4 >A 5 >A 1 >A 2 H=[0.4,0.5,0.1] MH=[0.2,0.7,0.1] M=[0,0.9,0.1] 9 VVH=[0.7,0.1,0.2] VH=[0.5,0.3,0.2] A 3 >A 4 >A 5 >A 1 >A 2 H=[0.3,0.5,0.2] MH=[0.1,0.7,0.2] M=[0,0.9,0.1] 10 VVH=[0.6,0.1,0.3] VH=[0.4,0.3,0.3] A 3 >A 5 >A 4 >A 1 >A 2 H=[0.2,0.5,0.3] MH=[0,0.7,0.3] M=[0,0.9,0.1] 11 VVH=[0.5,0.1,0.4] VH=[0.3,0.3,0.4] A 3 >A 4 >A 5 >A 1 >A 2 H=[0.1,0.5,0.4] MH=[0,0.7,0.3] M=[0,0.9,0.1] 12 VVH=[0.4,0.1,0.5] VH=[0.2,0.3,0.5] A 3 >A 5 >A 2 >A 4 >A 1 H=[0,0.5,0.5] MH=[0,0.7,0.3] M=[0,0.9,0.1] 13 VVH=[0.3,0.1,0.6] VH=[0.1,0.3,0.6] A 3 >A 5 >A 4 >A 1 >A 2 H=[0,0.5,0.5] MH=[0,0.7,0.3] M=[0,0.9,0.1] Step 7: Determne the fnal rankng of PPMIS The fnal score of each system was derved by calculatng the correspondng relatve closeness coeffcent wth respect to the ntutonstc fuzzy deal soluton. For each alternatve A, the relatve closeness coeffcent C wth respect to the IFPIS s defned as fol-

13 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems 103 Fg. 3. Screenshots of the method mplementaton n spreadsheets. (Colours are vsble n the onlne verson of the artcle; /IDT ) lows: S C = (11) S + S where 0 C 1. Equaton (11) was used to calculate these coeffcents (fnal scores) lsted n column (3) of Table 9. The alternatve PPMIS were ranked n a descendng order of these scores as A 3 >A 4 >A 1 >A 5 >A 2, from where t can be deduced that alternatve A 3 s the most domnant PPMIS for the present case study. Step 8: Senstvty analyss Senstvty analyss s concerned wth what-f knd of scenaros to determne f the fnal answer (rankng) s stable to changes (experments) n the nputs, ether judgments of alternatves or weghts of crtera. In the present case, senstvty analyss was frst performed by examnng the mpact of crtera weghts (.e., the weghts of users requrements from a PPMIS) on the fnal PPMIS rankng. Of specal nterest was to see f crtera weghts changes alter the order of the alternatves. 18 experments were conducted n a smlar way wth the approach presented n [5]. The detals of all experments are shown n Table 10, where w 1, w 2,...,w 13 denote respectvely the weghts of crtera IGLM, IE, PP1, PP2, PP3, PC1, PC2, PC3, PT1, PT2, AC, PO, CC. In Exps 1 5, weghts of all crtera were set equal to [0.10, 0.90], [0.25, 0.70], [0.50, 0.45], [0.75, 0.20] and [0.90, 0.10], respectvely. These IFNs correspond to the lngustc terms VU, U, M, I and VI, respectvely (see Table 2). In Exps 6 18, the weght of each of the 13 crtera was set equal to the hghest IFN [0.90,0.10], one by one, and the weghts of the rest of crtera were set all equal to the lowest IFN [0.10,0.90]. The results show that PPMIS A 3 remans the domnant alternatve n 14 out of the 18 experments (ths represents a clear majorty equal to 77.77%). PPMIS A 1 was frst n 2/18 experments, namely n Exps 10 and 17, where the hghest weghts were assgned, respectvely, to crteron PP3 (project plannng) and crteron PO (total prce for purchasng/ownershp). System A 2 had the hghest score n Exp. 6, where the hghest weght was assgned to crteron IGLM (Idea Generaton/Lead Management), whle system A 4 had the hghest score n Exp. 13, where the hghest value was assgned to the weght of PC3 (portfolo controllng). Further senstvty analyss on the fnal rankng can be performed by changng the IFNs presented n Table 2 and Table 3. For example, we can notce (n Table 4) that decson makers have utlzed specfc lngustc terms (.e., VVH, VH, H, MH and M) to express ther judgments on the performances of the alternatve PPMIS wth respect to the evaluaton crtera. Table 11 shows 13 addtonal experments appled to study the senstvty of the fnal rankng wth dfferent values of IFNs for the utlzed lngustc terms (VVH, VH, H, MH and M). Each experment s assocated wth a dfferent degree of hestaton. Table 11 presents the rankngs fnally produced by: ) consderng that hestaton degrees are all equal to zero (Exp. 1), ) ncreasng gradually the hestaton degrees and consderng that hestaton s subtracted from non-membershp (Exps 2 7), ) ncreasng gradually the hestaton de-

14 104 V.C. Geroganns et al. / Evaluaton of project and portfolo Management Informaton Systems grees and consderng that hestaton s subtracted from membershp (Exps 8 13). From Table 11, t can be seen that the best and worst PPMIS are not senstve to changng hestaton degrees. PPMIS A 3 was the most preferable alternatve n all experments, whle PPMIS A 2 was the least preferable alternatve n 12 out of the 13 experments. Thus, by applyng senstvty analyss we can conclude, wth a hgh confdence, that system A 3 s the most sutable PPMIS. Generalzaton and further valdaton of the presented approach requre the use of a fully parametersed form of the hestaton degree. Ths can be performed n two ways: ) by askng users/evaluators (decson makers) to express also a dfferent hestaton degree for each assessment, based on ether a Postve- Confdence or a Negatve-Confdence approach [33] or ) by askng users/evaluators to express ther judgments by utlzng nterval-valued ntutonstc fuzzy numbers [23]. We have plans to nvestgate these two solutons n a future research. In addton, we ntend to apply the decson makng approach n software selecton problems whch nvolve large number of stakeholders and decson makers. 7. Conclusons The paper presented, through a case study, the applcaton of a group-based mult crtera decson makng (MCDM) method for the evaluaton and fnal selecton of an approprate Project and Portfolo Management Informaton System (PPMIS). The appled method jontly syntheszed ntutonstc fuzzy sets and TOPSIS. The beneft from ths combnaton n a PP- MIS selecton approach s twofold: Frst, the approach actvely nvolves decson makers and PPMIS users n the decson makng process and aggregates ther opnons to support agreement upon the fnal selecton. Second, the approach consders that they both express ther judgments under nherent uncertanty. More sgnfcantly, the approach handles adequately the degree of ndetermnacy that characterzes both decson makers and users n ther evaluatons. Ths s very mportant when an organzaton needs to decde upon the selecton of any new, mult-functonal nformaton system, as n our case s a sutable PPMIS, snce decson makers often cannot have full knowledge of the extend that each canddate system wll (or wll not) support the user requrements. System users, on the other hand, can be unfamlar wth the processes supported by the requred system, and thus, they cannot judge wth certanty the mportance of ther needs. The presented approach not only valdated the method, as t was orgnally defned n [7], n a new applcaton feld that s the evaluaton of PPMIS (where other MCDM approaches are rather lmted n the lterature), but also consdered a more extensve lst of beneft and cost-orented crtera, sutable for PPMIS selecton. In addton, fnal results were verfed by applyng senstvty analyss. We should menton that the method underlyng computatons are not transparent to the problem stakeholders whch utlse lngustc terms to state evaluatons/preferences. Actually, we mplemented the method n a spreadsheet program that helps to effectvely and practcally apply the approach wth a varety of nputs. Example screenshots of ths spreadsheet are shown n Fg. 3. Fgure 3(a) presents an excerpt of user opnons on the mportance of the crtera (an excerpt of the nput data shown n Table 6). Fgure 3(b) presents an excerpt of the crtera weghts (an excerpt of the data shown n Table 7). Fgure 3(c) presents excerpts of: ) the aggregated ntutonstc fuzzy decson matrx (Table 5), ) the aggregated weghted ntutonstc fuzzy decson matrx (Table 8), ) the ntutonstc fuzzy postve deal and negatve deal solutons (step 5 of the method). The approach rases several ssues that could spark further research. For example, an nterestng dea could be to valdate the approach applcablty n addressng the selecton of other types of software packages. We are now nvestgatng the selecton of e-learnng management systems for the case organzaton (.e., the Hellenc Open Unversty). In addton, treatng more wth uncertantes would further strengthen the proposed approach n dervng more precse results. We have also plans to examne the utlzaton of more powerful methods n the same doman, such as the ntervalvalued ntutonstc fuzzy sets [12,23]. Acknowledgments The authors would lke to thank the anonymous revewers for ther helpful suggestons, as well as Ilas Magloganns, Lazaros Ilads and Harry Papadopoulos for ther knd nvtaton to partcpate n the specal ssue of the Intellgent Decson Technologes Journal. Ths paper s an updated and extended verson of an artcle presented n the 12th EANN/7th AIAI 2011 Conference. The research presented n ths paper has been co-fnanced by the European Unon (European Socal Fund) and Greek natonal funds through the Operatonal Program Educaton and Lfelong Learnng of the Natonal Strategc Reference Framework.