Performnce Evlution of Acdemic Librries Implementtion Model Luiz Bptist Melo CIDEHUS UE nd Librries of the Fculty of Science, University of Porto (Applied Mthemtics nd Botnicl Deprtments) Ru do Cmpo Alegre, 687, 469-007 PORTO Portugl- E-mil: lbmelo@fc.up.pt Cesltin Pires CEFAGE-UE nd Mngement Deprtment of University of Évor Lrgo dos Colegiis, nº 2, 7000-803 ÉVORA Portugl- E-mil: cpires@uevor.pt Abstrct This pper describes n ction model to mesure the performnce of librries. The proposed model is bsed in the CAF Common Assessment Frmework, the Blnced Scorecrd nd the Anlytic Hierrchy Process (AHP). The recommended performnce indictors re bsed in the ISO 620:998, Adm:2003 nd ISO 2789:2003. The reltive weights for ech performnce mesure re clculted using the AHP. The AHP computes the weights of the performnce mesures in two steps: Compring (pirwise) the performnce mesures under ech criterion; Compring (pirwise) the mjor criteri of the proposed model. The min purpose of this model is to improve the performnce of librries nd to develop benchmrking techniques. The model ws tested using the opinion of ten university librrins. According to the dt, the most importnt criteri, by decresing order of importnce, re: customer stisfction, impct on society, ledership nd finncil perspective. Key Words: Acdemic librry, evlution, ssessment, service qulity, CAF, Blnced Scorecrd, Anlytic Hierrchy Process Introduction Acdemic librries services hve chnged very fst in the lst twenty yers. Nowdys, electronic resources, networks nd the World Wide Web represent lrge frction of the librry nd informtion services. Acdemic librries must lso be ble to demonstrte the vlue of wht they re doing nd provide evidence of the impct tht they re mking (Pyne, 2005), so tht it is importnt to use methods to find ssessment dt. Librrins must mnge stff, informtion in severl supports, nd technicl ctivities to
2 produce qulity services. Qulity services men resources nd services tht stisfy the user s expecttions (Prsurmn et l., 985). This rticle proposes frmework to evlute the performnce of cdemic librries. This frmework is quite verstile nd it cn esily be djusted to be implemented in other types of librries (public librries, university librries documenttion nd informtion centers). The purpose of this study is to present model to mesure the contribution of the cdemic librry to the institution where it belongs nd to the society. The finl output of our model is globl performnce mesure. To compute this globl performnce mesure, the institution hs to choose the set of criteri tht should be tken into ccount in the performnce evlution nd their reltive weights. In ddition, the institution needs to choose the set of performnce indictors tht should be used to mesure ech criterion nd their reltive importnce in tht mesure. As it will becme cler ltter, the dilogue mong the stff, the users of the cdemic librry, nd externl librrins is extremely importnt in the ssessment process. The choice of criteri nd indictors nd the weights given to ech criterion or to ech indictor re the outcome of prticiptive performnce evlution process. Our model proposes set of criteri nd indictors which re relevnt to mesure performnce in cdemic librries nd it proposes method which cn be used to determine the weights of ech criterion or indictor. The proposed set of criteri is bsed on the Common Assessment Frmework (CAF) (Europen Institute of Administrtion, 2006) nd the Blnced Scorecrd (BSC) (Kpln & Norton, 992). The set of criteri includes enblers nd results (s define in CAF). In ddition, it ssumes the four Blnced Scorecrd dimensions: user perspective, internl process perspective, finncil perspective nd lerning/growth perspective. The recommended performnce indictors re bsed on the ISO 620:998 (Interntionl Orgniztion for Stndrdiztion, 998), Adm:2003 (Interntionl Orgniztion for Stndrdiztion, 2003), nd ISO 2789:2003 (Interntionl Orgniztion for Stndrdiztion, 2003b). To determine the weights of ech criterion or indictor, our innovtive model resorts to the Anlytic Hierrchy Process (AHP) (Sty, 990). This process helps us to determine the reltive importnce of ech performnce indictor (in ech criterion mesurement) nd the reltive importnce of ech criterion (in the globl performnce evlution). The pper is orgnized s follows: In the next section we present n overview of the CAF nd Blnced Scorecrd models nd the Anlytic Hierrchy Process. In the following section we describe the different perspectives define in the proposed model supported in CAF nd BSC nd the corresponding ssocited performnce indictors. Section 4 describes the implementtion of the model nd discusses the results. Finlly, we present the conclusions.. The Building Blocks of the Model: n overview. CAF Common Assessment Frmework The first version of the CAF ws presented in My 2000 during the First Europen Qulity Conference for Public Administrtions held in Lisbon. This mngement qulity tool is bsed on the Europen Foundtion for Qulity Mngement Excellence Model (Europen Foundtion for Qulity Mngement, 2006) tht is composed of nine criteri. The set of enbler s criteri includes: ledership, humn resources mngement,
3 policy nd strtegy, externl prtnerships nd resources nd process nd chnge mngement. The set of results involves: people results, customer oriented results, impct on society nd key performnce results. The ssessment of ech criterion develops into severl sub-criterions. The globl CAF structure is composed of 32 issues. These issues re the sub-criterions tht re systemticlly evluted using scle, with six points, illustrted in tble, for enblers, nd tble 2 for results (Europen Institute of Public Administrtion, 2006). Tble - Assessment Scle of the Model CAF for Enblers (Europen Institute of Public Administrtion, 2006) Score Enblers 0 No evidence or only necdotl evidence of n pproch. An pproch is plnned P (pln). 2 An pproch is plnned nd implemented D (do) 3 An pproch is plnned implemented nd reviewed C (check). 4 An pproch is plnned implemented nd reviewed on the bsis of benchmrking dt nd djusted ccordingly A (ct). 5 An pproch is plnned implemented reviewed on the bsis of benchmrking dt djusted nd fully integrted into the orgnistion. Tble 2 - Assessment Scle of the Model CAF for Results (Europen Institute of Public Administrtion, 2006) Score Results 0 No results re mesured. Key results re mesured nd show negtive or stble trends. 2 Results show modest progress. 3 Results show substntil progress. 4 Excellent results re chieved nd positive comprisons to own trgets re mde. 5 Excellent results re chieved positive comprisons to own trgets re mde nd positive benchmrks ginst relevnt orgnistions re mde. It is interesting to observe tht this scle identifies the PDCA circle (Bnk, 998) creted by W. Edwrd Deming, in 950. The mening of the PDCA circle is pln, do, check nd ct.
4.2 Blnced Scorecrd (BSC) Blnced Scorecrd (Kpln & Norton, 992) is performnce mngement model to ssess orgniztions. This tool expresses the orgniztion s strtegy nd vision using four importnt issues: finncil perspective, internl process perspective, innovtion nd lerning perspective nd customer perspective (Kpln & Norton 996).This evlution technique estblishes blnce between orgniztion vision, performnce indictors, gols nd implementtion ctions versus finncil perspective, internl process perspective, innovtion nd lerning perspective nd customer perspective tht cn be represented in scorecrd (Kpln & Norton, 2004)..3 Anlytic Hierrchy Process (AHP) The Anlytic Hierrchy Process (AHP), developed by Thoms L. Sty (990), is multiple criteri decision technique. When decision mker hs to choose mong severl lterntives nd there re multiple criteri which re relevnt for the decision mker, it my be quite difficult for the decision mker to estblish the reltive importnce of ech criterion nd/or to evlute how good is ech lterntive reltively to ech criterion. The AHP is methodology which helps the decision mker in determining the vlue of ech lterntive, nd the weights of ech criterion. There re thousnds of pplictions of AHP to complex decision problems (Sty, 990; Anderson, D.R. et l.; 998; Sty, 2000). It turns out tht the AHP cn be useful in the librry performnce mesurement. The min purpose is to find the weight of ech performnce indictor using the AHP. The AHP computes the weights of the performnce mesures in two steps ( Anderson, D.R. et l., 998):. The first step designs hierrchic structure (see figure ), with the vrious criteri used in the performnce evlution nd the ssocited performnce indictors. 2. The second step determines the weights t ech hierrchy level. The nlysis strts t the lower hierrchy level (determine the weight of ech indictor in the evlution of certin criterion). Next the upper hierrchy level is nlyzed (determine the weight of ech criterion in the globl performnce mesure). GLOBAL PERFORMANCE MEASURE D CRITERIA (st level) B B 2 B m C E F PERFORMANCE INDICATORS C 2.... E 2.... F 2.... (2nd level) C n E r F s Figure - Hierrchy Anlyse of the problem D The technique proposed by Sty hs two very importnt insights: (i) the ide tht decomposing the globl problem into hierrchy of simpler problems simplifies the tsk of nlyzing the problem. (ii) The ide tht it is esier to compre pirwise criteri
5 (or indictors) thn to give weights. This explins why the second step is bsed on pirwise comprisons. Lower hierrchy level study determining the weight of ech indictor in given criterion The lower hierrchy level study hs the following steps: Construction of pirwise comprison mtrix; Synthesis; nd Consistency rtio estimtion. Pirwise comprison mtrix To find the weights of ech criterion/indictor it is necessry to construct mtrix providing judgments bout the reltive importnce between pirs of elements in ech hierrchy (see tble 3). Tble 3 Criteri Pir Comprison Mtrix concerning AHP for criterion B C C 2... C n C = C 2 2 2 2...... n 2 n............ C n n = n n 2 = 2 n... Sty estblished scle to compre ech pir of criteri/indictor, the scle proposed is represented in tble 4 (Anderson, D.R. et l., 998). Tble 4 Pirwise comprison scle for AHP preferences used in confront between the criteri nd performnce indictors ( Anderson, Sweeney & Willims, 998 Judgment of preferences Numericl Rting Extremely preferred 9 Very strongly to extremely 8 Very strongly preferred 7 Strongly to very strongly 6 Strongly preferred 5 Modertely to strongly 4 Modertely preferred 3 Eqully to modertely 2 Eqully preferred Tble 5 represents the performnce indictors pir comprisons mtrix concerning AHP for criterion B, where ij is the quntittive judgment between the performnce
6 indictor C i compred with the performnce indictor C j nd re define the following rules: ) If ij = α, then b) ii =, for ll i. ji = / α, α 0 This procedure is repeted for ll the other criteri (B 2, B 3,, B m ). Tble 5 The normlized pirwise comprison mtrix to the criterion B concerning AHP methodology C C 2... C n C C 2 = n i 2 2 = n i 2 2 = n i 2 22 = n i 2...... n 2n............... C n n n = n i = n2 n n2... i 2 n = n = n 2n in in nn = n in Synthesis After developing the mtrix of pirwise comprisons, we cn estimte the reltive weigh of ech performnce indictor. This estimtion is done in three steps: - Estimte the totl of ech column of the pirwise comprison mtrix; 2 - Divide ech element in the pirwise comprison mtrix by its column totl. The resulting mtrix is the normlized pirwise comprison mtrix (see tble 5). Tble 6 - RI vlues re function of the number of elements being compred n (Anderson, Sweeney & Willims, 998) 2 3 4 5 6 7 8 9 0 0 0 0,58 0,90,2,24,32,4,45,5 3 - Clculte the verge of the elements in ech row of the normlized pirwise comprison mtrix. The result is the priority vector tht gives the reltive weight of ech performnce indictors in the mesurement of criterion B. The priority vector hs the following form:
7 ω ω2... ω n Where ω i is the weight of the performnce indictor C i nd it is equl to the verge of the elements in row i of the normlized pirwise comprison mtrix. n j = ij ω i = () n Using these weights, we cn now estimte the vlue of criterion B : B = ω i C i (2) This process is repeted for ll estblished criteri (B, B 2,..., B m ). Consistency rtio estimtion n The AHP recommends the estimtion of vlue clled the consistency rtio (Anderson, D.R. et l., 998; Sty, 2000). The consistency rtio mesures the consistency of the pirwise comprison judgments. In other words, this step evlutes the qulity of the judgments estblished in the pirwise comprison mtrix. Since the finl weights depend on the pirwise comprison mtrix, it is importnt to gurntee tht the degree of inconsistency mong the pirwise comprisons is reltively smll. It is dmitted tht rtio exceeding 0.0 is indictive of inconsistent judgments. Vlues of consistency rtio of 0.0 or less re considered resonble. When the consistency rtio is bove 0.0, the pirwise comprisons mtrix needs to be re-evluted so s to reduce the degree of inconsistency. The consistency rtio estimtion develops in four steps ( Anderson, D.R. et l., 998): Multiply ech column element of the pirwise comprison mtrix by the reltive priority of ech item considered. Sum the vlues cross the rows to clculte the weighted sum vector.... n 2 2... n2......... n 2n... ω ω2 =... ω n w w... w 2 n n = j w j j = n = j w j j = 2 n = j w j j = Weighted sum vector 2 Divide the vlue of ech weighted sum by the corresponding priority vlue. The verge of these vlues is define s λ mx. n w j j= ω j λ = mx (3) n 3 - Clculte the consistency index (Sty, 990), which is defined s: ( λmx n) CI = (4) n 4 - The consistency rtio (CR) is computed by:
8 CI CR = (5) RI where RI Rndom index, is the consistency index of rndomly generted pirwise comprison mtrix. The vlues of RI depend on the number of elements being compred (see tble 6). As mentioned before, consistency rtio bove 0.0 indictes unresonble inconsistency in the pirwise comprisons. Thus this mtrix hs to be re-evluted nd the procedure hs to be repeted till the consistency rtio is cceptble. Upper hierrchy level study determining the weights of ech criterion in the globl performnce mesure The upper hierrchy study estimtes the Globl performnce mesure (D) s weighted verge of the criteri (B, B 2, B 3,..., B m ). This estimtion involves three steps (construction of pirwise comprison mtrix, synthesis, nd consistency rtio estimtion) nd it is similr to the lower hierrchy level study. In summry, the Globl performnce mesure (D) (Melo, 2005) is estimted tking into ccount nd the vlue of ech criterion. q q2... q m the priority vector D = q B + q 2 B 2 +... + q m B m (6) m D = q i B i (7) 2. Selecting the criteri nd performnce indictors 2. Working with both models CAF nd BSC As mentioned bove, the set of criteri proposed in our model is bsed simultneously in: - The Common Assessment Frmework (CAF) (Europen Institute of Public Administrtion, 2006), which hs been used to evlute the services performnce in the Europen Union Public Administrtion; nd, - The Blnced Scorecrd (BSC) (Kpln & Norton, 992), which hs been used in severl projects to ssess cdemic librry performnce, such s: Librry University of Münster, in Germny (Poll, 200), nd for public librries in Crlsbd City, CA, United Stts of Americ (Mtthews, 2005). We decided to use n integrted model, which cptures the dvntges of BSC perspectives nd CAF criteri. This model, is n dpttion from BSC/EFQM (Andersen, Lwrie & Shulver, 2000). The reltionship between CAF nd BSC mkes our method stronger becuse it brings to the ssessment process the following issues: The librry s blnced scorecrd trnsltes the interests of the institution by exmining the librry strtegy nd vision using four perspectives which represent the different fcets of the performnce linked together by cuse nd effect. Since different librries my hve different visions nd strtegies, the outcome of this
9 process depends on the librry being evluted. On the contrry, CAF is independent of the librry, the set of predefined criteri nd mesurement res re the sme for ll the orgniztions. This model provides dt to use benchmrking techniques; Blnced scorecrd identifies set of priorities tht the librrin tem believes will deliver strtegy. CAF gives comprehensive description nd ssessment of how processes runs into the librry nd re mnged nd performnce; The BSC is built round vision for wht the librry wnts to chieve 2-5 yers in future. CAF outcomes describes the current stte of the librry s process; The BSC model does not serch the qulity of processes nd ctivities tody. On the other hnd CAF encourges continuous improvement cross services nd resources librry; nd, Environment nd impct on the society is n explicit prt of CAF model nd therefore ny self-ssessment is going to provide feedbck on this dimension, however BSC does not ssume systemticlly these dimensions. 2.2 Criteri nd performnce indictors The implementtion of performnce evlution model is not n esy tsk. It is necessry to brek certin concepts nd inerti nd to stimulte tem work. One of the purposes of the performnce evlution is to involve the vrious elements of the institution in the functionl nd process nlysis of the orgniztion. A benefit of this procedure is the exchnge of knowledge between ll the prticipnts nd between similr orgniztions (Cullen, 2003). The performnce evlution progrm is successful if it provides prcticl mngement orgniztionl model tht leds to continuous improvement chnge. Our model hs gret flexibility in the evlution scle becuse it is bsed on CAF criteri (Europen Institute of Public Administrtion, 2006) nd Blnced Scorecrd perspectives (Kpln & Norton, 992). All proposed criteri nd perspectives re converted into the respective performnce indictors, defined for ISO620:998, dm:2003 (Interntionl Orgniztion for Stndrdiztion, 2003) nd ISO 2789:2003 (Interntionl Orgniztion for Stndrdiztion, 2003b). 3. Model implementtion 3. Methodology In order to show how our model cn be implemented we conducted n inquiry to ten cdemic librrins (nine Portuguese nd one Brzilin), from different universities (Universidde Ctólic Portugues - Brg, Universidde do Porto, Universidde Lusíd de Fmlicão, Instituto Politécnico do Porto, Universidde Nov de Lisbo, Universidde do Algrve, Universidde d Mdeir nd Universidde dos Açores from Portugl nd Universidde de São Pulo from Brzil). These ten librrins performed pirwise comprisons between the vrious performnce indictors of ech criterion, nd pirwise comprisons between the vrious criteri. In other words, ech librrin hd to indicte his/her own pirwise comprison mtrices. The opinions of the ten librrins were ltter on ggregted (for ech pirwise comprison we computed the verge of the ten librrin vlues). The vrious performnce indictors verge pirwise comprison mtrices re the following (to the lower hierrchy level study determining the weight of ech indictor in given criterion):
0.000 0.254 0.254 3.940.000 0.366 3.934 2.734.000 Ledership.000 4.220 4.700 Strtegy nd plnning 0.237.000 3.000 0.23 0.333.000.000 0.226 0.92 4.420.000 0.3 5.220 3.220.000 Externl prtnerships nd resources.000 4.64 5.300 Process nd chnge mngement 0.27.000 3.40 0.89 0.38.000.000 0.22 0.49 0.84 0.232 4.720.000 0.243 0.34 0.366 6.74 4.4.000 0.52 0.446 5.429 2.929.920.000 0.452 4.34 2.729 Customer 2.244 2.2.000.000 0.22 0.269 0.256 4.74.000 0.400 0.400 3.74 2.500.000 0.400 3.9 2.500 2.500.000.000 Impct on society 0.556 0.207.800.000 0.248 4.833 Finncil 4.033.000 The criteri verge pirwise comprison mtrice is the following (to the upper hierrchy level study determining the weights of ech criterion in the Globl performnce mesure (D)):.000 3.000 0.333.000 0.200 0.333 0.250 0.500 7.000 5.000 5.000 2.000 0.333 3.000 5.000 3.000.000.000 4.000 3.000 0.500 4.000 0.43 0.200 3.000 2.000 0.200 0.500 0.333.000 0.250 0.333 2.000.000 0.250 3.000.000 4.000.000 0.333 0.333 0.333 3.000.000 0.333.000 3.000 3.000.000 Using the ggregted pirwise comprison mtrices, we finlly computed the weights tht reflect the group opinion, using the AHP. Tble 7, presents the reltive weights, for ech performnce perspectives to cdemic librries, computed with the mixed-methods CAF-BSC nd AHP (Melo, 2005). Tble 8 indictes the reltive weights for ech performnce indictor, consistency rtios nd reltive weight of ech performnce indictor on the Globl performnce mesure (D) clculted with the sme methodology. Figure 2 uses the hierrchic structure of the AHP nd shows how, the inquired librrin, weight the different criteri used to compute the Globl performnce mesure (D), nd how they weight the vrious performnce indictors used to evlute ech criterion. The clcultions were estimted with the softwre Excel for Windows. Tble 7 The reltive weights of the performnce perspective mesures to cdemic librries Perspective Reltive Weight Ledership 0.82 Strtegy nd plnning 0.076 Externl prtnerships 0.069 Process nd chnge mngement 0.00 Customer 0.28 Impct on society 0.94 Finncil 0.6
Tble 8 The reltive weights of the performnce indictors to cdemic librries, consistency rtio estimtion nd reltive weight of performnce indictor on D Performnce Indictor (PI) Reltive Weight in the criterion (PI) Consistency Rtio (PI) Reltive weight of the Perspective mesure Consistency Rtio criteri Reltive weight of performnce indictor on D M. The development nd formultion of vision nd mission (wht re our gols) of the librry 0.640 0.6 M.2 The development of n orgnistionl structure in ccordnce with the tsks of librry 0.0996 0.82 0.238 0.043 M.3 The promotion nd trining to improvement the ctivities 0.23 0.220 M.4 The systemtic gthering informtion bout the needs nd expecttions of the users M.5 The reorgnistion nd improvement strtegies nd methods of librry ctivities 0.663 0.050 0.229 0.0966 0.076 0.07 M.6 The development nd pplying methods to mesure the librry performnce nd benchmrking 0.07 0.008 M.7 Number of monogrphs. e-books journl nd e-journls M.8 Opening hours for week M.9 The identifiction of strtegic prtners nd the 0.677 0.047 0.225 0.06 0.099 0.069 0.098 0.007 nture of the reltionships M.0 Medin time of document cquisition M. Medin time of document retrievl from Open Access Are 0.684 0.068 0.28 0.0998 0.00 0.022 0.0038 M.2 To ssess mrket penetrtion of electronic service 0.098 0.00 R. Librry visits per cpit 0.52 0.4 R.2 Lons per cpit R.3 Overll user stisfction (between 0-5) R.4 Overll stff stisfction (between 0-5) 0.22 0.046 0.08 0.0962 0.28 0.024 0.092 0.020 R.5 Levels of bsenteeism or sickness 0.068 0.05 R.6 Rte of the students success (for exmple number of students tht get the grdution per yer) R.7 Rte of professors nd reserchers publiction R.8 Rte of the stff librry prticiption in internl discussion groups. interntionl professionl meetings with senior Librrins. etc. 0.545 0.06 0.24 0.042 0.50 0.0998 0.94 0.029 R.9 The degree of complince with environmentl principles (for ex. The energy sving. the reduction of wste nd pckging. the use of recycled mteril) 0.09 0.08 R.0/R. Cost per user (professor / student) R. 2 Cost per librry visit 0.549 / 0.35 0.088 / 0.057 0.059 0.6 0.00 0.06
2 Globl Performnce Mesure (D) CRITERIA st level Ledership 0,82 Strtegy & Plnning 0,076 Externl Prtnership & Resouces 0,069 Process nd chnge mngement 0,00 Client 0,28 Impct on Society 0,94 Finnce 0,6 PERFORMANC E INDICATORS 2nd level M. 0,640 M.2 0,238 M.3 0,23 M.4 0,663 M.5 0,229 M.6 0,07 M.7 0,677 M.8 0,225 M.9 0,098 M.0 0,684 M. 0,28 M.2 0,098 R. 0,52 R.2 0,22 R.3 0,08 R.4 0,092 R.5 0,068 R.6 0,545 R.7 0,24 R.8 0,50 R.9 0,09 R.0 0,549 R. 0,35 R 0,00 Enblers Results Figure 2 - Hierrchic structure of the AHP nd results how, the inquired librrin, weight the different criteri used to compute the Globl performnce mesure (D), nd how they weight the vrious performnce indictors used to evlute ech criterion. 3.2 Discussion of the results The Brzilin librrin Smpio (2005), mentioning Piget, ffirms tht coopertion is n ction process where the ccomplishment is executed in common. For n effective coopertion the following conditions re necessry: existence of common scle of vlues, conservtion of the vlues' scle nd existence of reciprocity in the interction. This reserch involved coopertion mong ten cdemic librrins from severl universities. This study proposed severl criteri nd performnce indictors to evlute cdemic librries nd computed the vlues of the reltive weights for ech performnce mesure, bsed on the opinion of cdemic librrins. We inquired nine librrins of lrge frction of Portuguese universities. In ddition, we highlight the prticiption of one Brzilin librrin from the University of São Pulo. Most contcts were performed by e-mil, there were severl contcts during period of three months (November nd December 2006 nd Jnury 2007). The inquiry ws vilble in n e-mil ttchment. Our results indicte tht there re four criteri which re considered prticulrly importnt in performnce evlution (see tble 7 nd figure 2): customer perspective (weight 0.28), impct on society (weight 0.94), ledership (weight 0.82) nd finncil perspective (weight 0.6). On the other hnd, process nd chnge mngement (weight 0.), strtegy nd plnning (weight 0.076), nd externl prtnerships nd resources (weight 0.069) re considered reltively less importnt. As we explined bove, ech criterion is computed s weighted verge of severl performnce indictors. Tble 8 shows the weights of ech performnce indictor in given criterion (second column of tble 8).
3 Customer perspective is the most importnt criterion in performnce evlution, with weight of 0.28. Customer perspective involves the performnce indictors: librry visits per cpit, lons per cpit, user stisfction, stff stisfction nd levels of bsenteeism nd sickness. According to the inquired librrins the most importnt indictor in the customer criterion is librry visits per cpit, with weight of 0.52. The second most importnt criterion is the Impct on society, with reltive weight of 0.94. This dimension is relted with the different librry outcomes. The two most importnt indictors for this criterion re: the rte of students success (with weight of 0.545) nd the rte of professors nd reserchers publictions (with weight of 0.24). This dt is consistent with Poll nd Pyne ffirmtions (2006), cdemic outcomes supports the following gols; effective teching (high grdution rtes, high grdes in exmintions ) effective reserch (high renown nd use of reserch results nd publictions...). Ledership is the third most importnt criterion, with weight of 0.82. In this criterion the most importnt indictor is, by fr, the development nd the formultion of the vision of the librry (reltive weight of 0.640). Finncil perspective is lso considered n importnt spect for cdemic librrins, with reltive weight of 0.6. In this criterion the cost per user (professor/ student) re the most importnt indictors. Process nd chnge mngement criterion hs reltive weight of 0.00. For this criterion, the indictors re: the medin time of document cquisition (weight of 0.684), medin time of document retrievl from open ccess re (weight of 0.28) nd the mrket penetrtion of electronic services (weight 0.098). Consequently, the performnce indictor medin time of document cquisition is extremely importnt. The dimension Strtegy nd plnning, with weight of 0.076, nd Externl prtnerships nd resources (with weight 0.069) presented similr vlues. We highlight tht the most importnt ssocited performnce indictors re, the systemtic gthering informtion bout the needs nd the expecttions of the users (weight 0.663) nd the collection of pper nd electronic resources development (weight 0.677), respectively. The consistency rtios, for the severl criteri nd ssocited performnce indictors, present vlues between 0.0038 nd 0.0998 (columns 3 nd 5 of tble 8). Since the consistency rtios re ll below 0.0, one cn conclude tht the judgments estblished, by ech librrin, in the pirwise comprison mtrices, were internlly consistent. Since the globl performnce mesure is weighted verge of the vrious criteri, nd ech criterion is weighted verge of severl performnce indictors, we cn lso compute the globl performnce mesure s weighted verge of the vrious performnce indictors. The weight of ech performnce indictor on the Globl performnce mesure D, is equl to the product of the criterion weight by the performnce indictor weight on the criterion. Column 6 in tble 8, presents the weight of ech performnce indictor on the globl performnce mesure D nd figure 3 grphs these weights by decresing order of importnce. This nlysis shows tht, overll, the three most importnt performnce indictors re:
4 - The development nd formultion of vision nd mission of the librry. The reltive weight of this performnce indictor on D is 0.6; - Librry visits per cpit (the reltive weight of performnce indictor on D is 0.4; nd - Rte of the student success, for exmple number of students tht get the grdution per yer, (reltive weight of performnce indictor on D is 0.06). This tem of librrins ssumed tht cost per professor nd investigtor is more importnt thn cost per student (respectively 0.088 nd 0.057). In ddition, they think tht the following issues re still importnt: medin time document cquisition (0.068), the systemtic gthering informtion bout the needs nd expecttions of the users (0.050), number of books nd journls in pper nd electronic versions (0.047), nd lons per cpit (0.046). In order to relize high-qulity service, librrins need to understnd how users (students, techers nd investigtors), stff nd society recognize nd evlute university librry services. This reserch ws n excellent exercise to enhnce discussion, mong the Portuguese librrins, methodologies, theories nd models to build frmework with 23 items; 2 performnce indictors for enblers nd performnce indictors for results. The methodology presented in this work ws lredy used to recognize best prctice in Portuguese Higher Eduction Librries (Melo et l., 2008). Reltive Weights of the Criteri Customer Impct on society Criterion Leder shi p Fi nnci l Pr ocess nd chnge mngement Str tegy nd pl nni ng Exter nl pr tner ships nd r esour ces 0 0,05 0, 0,5 0,2 0,25 Reltive Weight Figure 2 Reltive weights of the perspectives (or criteri)
5 Reltive Weights of the Performnce Indictors in the globl performnce mesure Development nd formultion of vision nd mission Librry visits per cpit Rte of the students success Cost per techer Medin time of document cquisition Cost per student Systemtic gthering informtion bout needs nd expecttions of users Number of monogrphs, e-books, journl nd e-journls Performnce Indictor Lons per cpit Development orgnistionl structure in ccordnce with the tsks Rte of professors nd reserchers publiction Rte stff librry prticiption in internl discussion groups, interntionl meetings Overll user stisfction Promotion nd trining to improvement the ctivities Medin time of document retrievl from Open Access Are Overll stff stisfction Degree of complince with environmentl principles Reorgnistion nd improvement strtegies nd methods ctivities Cost per librry visit Opening hours for week Levels of bsenteeism or sickness Assess mrket penetrtion of electronic service Development nd pplying methods to mesure the performnce Identifiction of strtegic prtners nd the nture of the reltionships 0,000 0,020 0,040 0,060 0,080 0,00 0,20 0,40 Reltive Weight Figure 3 - The weight of ech performnce indictor on the globl performnce mesure D by decresing order of importnce
6 CONCLUSIONS The evlution model proposed in this study is bsed on well-known instruments, CAF - Common Assessment Frmework, Blnced Scorecrd nd Anlytic Hierrchy Process. The min vlue dded of this rticle is to relte these three methodologies nd suggest prcticl evlution performnce tool which includes the benefits of these three instruments. This project recognizes tht mesuring cdemic librries inputs, outputs, processes nd outcomes is not n esy tsk nd it is gret chllenge. However our reserch lso shows tht the discussion of these subjects mong librrins is useful nd llows us to obtin dt for the evlution of librries nd informtion services.. This inititive hs demonstrted tht it is possible to build prtnerships to crete projects, to shre knowledge nd dt, to get performnce evlution results so s to improve qulity services in university librries. Bodi nd Mier-O She (2005) ffirm tht Librries hve to invest in preprtion for digitl future while mintining collections nd services bsed on predomintely print world. Mngement performnce theories re considertions importnt to the ssessment service qulity nd to overtke the new chllenges tht exist in the cdemic librries.. This study could be contribution to stff librry becuse it presents the following benefits suggested by Mtthews (2006): Clrify, updte nd communicte strtegy Link strtegic objectives to performnce mesures with ssocited long term trgets Broden mngements focus issues tht ffect sustinble long-term performnce Provide focus for continuous process improvement efforts nd qulity enhncement inititives Identify nd lign strtegic inititives Demonstrte ccountbility. After ll the purpose is to dpt the "librries to the Mn's mesure (Eco, 983). Acknowledgement The uthors wish to express their deep precition to the librrins Aid Alves (U. Ctólic Portugues), Helen Brbos (U. do Porto), Helen Mirnd (U. do Porto), Olind Mrtins (U. Lusíd - Fmlicão), Irene Mesquit (Instituto Politécnico do Porto) Lourdes Gouvei (U. Nov de Lisbo), Emíli Pcheco (U. do Algrve), Iolnd Silv (U. d Mdeir), An Tveir (U. dos Açores), e Imculd Smpio (U. São Pulo - Brzil), who generously took the time to prticipte in our focus group inquiries. REFERENCES Andersen, H., Lwrie, G. nd Shulver, M. 2000. The blnced scorecrd vs. the EFQM business excellence model, 2GC Working Pper, June, pp. -4. Anderson, D.R., Sweeney nd D.J., Willims, T.A. 998. Quntittive methods for business. 7 th ed. Cincinnti, Ohio: South-Western College. Bnk, J. 998. Qulidde totl: mnul de gestão. Mem Mrtins: CETOP. Bodi, S. nd Mier-O'She, K. 2005. The librry of Bbel: mking sense of collection mngement in Post-modern word, The Journl of Acdemic Librrinship, 3, pp.43-50. Cullen, R. 2003. Benchmrking: overview nd context [online], in: Proceedings of World Librry nd Informtion Congress 69th IFLA Generl Conference nd Council,
7 Berlin, -9 August. [Accessed 09 September 2003]. Avilble from World Wide Web: <http://www.ifl.org/iv/ifl69/ppers/05e-cullen.pdf> Eco, U. 983. A bibliotec. Lisbo: Difel. Europen Institute of Public Administrtion 2006. CAF Common Assessment Frmework. Mstrich: EIPA. [Accessed 0 October 2006]. Avilble from World Wide Web: <http:// www.eip.nl/caf/caf_2006/brochures/english_2006.pdf> Interntionl Orgniztion for Stndrdiztion 998. ISO 620 informtion nd documenttion - librry performnce indictors. Genève: ISO. Interntionl Orgniztion for Stndrdiztion 2003. ISO 620 informtion nd documenttion - librry performnce indictors: mendment : dditionl performnce des bibliothèques. Genève: ISO. Interntionl Orgniztion for Stndrdiztion 2003b. ISO 2789 informtion nd documenttion - interntionl librry sttistics. Genève: ISO. Kpln R.S. nd Norton, D.P. 992. The blnced scorecrd - mesures tht drive performnce. Hrvrd Business Review, Jnury Februry, pp. 7-79. Kpln, R.S. nd Norton, D.P. 996 The blnced scorecrd: trnslting strtegy into ction. Boston: Hrvrd Business School. Kpln R.S. nd Norton, D P. 2004. Kpln e Norten n prátic. Rio de Jneiro: Elsevier. Mtthews, J. R. 2005. Scorecrd for results: guide for developing librry Blnced Scoerecrd. Crlsbd, CA: The Crlsbd City Librry. Melo, L.B. 2005. Avlição de desempenho ds bibliotecs d Universidde do Porto: modelo de ctução. Évor, Inf. Sc. Ms. Disserttion - Universidde de Évor. Melo, L.B., Pires, C.nd Tveir, A. 2008 A. Recognizing best prctice in Portuguese Higher Eduction Librries. IFLA Journl, 34(), pp.34-54. Prsurmn, A., Zeithml, V.A. nd Berry, L.L. 985. A conceptul model of service qulity nd its implictions for future reserch. Journl of Mrketing, 9, pp.4-50. Pyne, P. 2005. Conyers, A. Mesuring the impct of higher eduction librries: the LIRG/SCONUL impct implementtion inititive. Librry nd Informtion Reserch, 29(9), [Accessed 29 October 2007]. Avilble from World Wide Web: <http://eprints.bbk.c.uk/48//pyne.pdf> Poll, R. 200. Performnce, process nd costs: mnging service qulity with blnced scorecrd. Librry Trends, 49(4), pp. 709-77. Poll, R. nd Pyne, P. 2006. Impct mesures for librries nd informtion services. Librry Hi Tech. 24(4), pp. 547-562.
8 Sty, T.L. 990. Decision mking for leders: the nlytic hierrchy process for decisions in complex world. Pittsburgh: University of Pittsburgh. Sty, T.L. 2000. Fundmentls of Decision Mking nd Priority Theory with Anlythic Hierrchy Process. Pittsburgh: RWS. Smpio, M. I. 2005. Motivção no trblho coopertivo: o cso d Rede Brsileir de Bibliotecs d Áre de Psicologi - ReBAP. São Pulo, Inf. Sc. Ms. Disserttion - Universidde de São Pulo.