PAOLO BELLINI, GIM Statistics and Evaluation

Transcription

1 Date submitted: 02/06/2009 Traditional indicators and multivariate statistical analysis: trends in Italian university libraries PAOLO BELLINI, GIM AUTHORS: GIM - Gruppo Interuniversitario per il monitoraggio e la valutazione dei sistemi universitari: Beatrice Catinella, Università degli Studi di Padova; Wanna Manca, Università degli Studi di Trento; Paolo Bellini, Università degl i Studi di Perugia; Maurizio di Girolamo, Luisanna Saccenti, Federica De Toffol, Università degli Studi Milano Bicocca; Eleonora Giusti, Francesca Landi, Università degli Studi di Firenze; Luca Bardi, Annalisa Mariani, Politecnico di Milano; Danilo Deana, Anna Maria Bellia, Università degli Studi di Milano; Marina Gorreri, Fiammetta Mamoli, Elisabetta Sparacio, Università degli Studi di Parma; Antonio Scolari, Università degli Studi di Pavia; Nunzia Spiccia, Maria Vittoria Savio, Politecnico di Torino; Mirella Mazzucchi, Serena Spinelli, Laura Bertazzoni, Università degli Studi di Bologna; Meeting: 21.6 Statistics and Evaluation WORLD LIBRARY AND INFORMATION CONGRESS: 75TH IFLA GENERAL CONFERENCE AND COUNCIL August 2009, Milan, Italy ABSTRACT During the few last years, the measurement and evaluation of library services in many italian universities has been largely driving thedevelopment of the organization and the establishment of university library systems (SBA Sistemi Bibliotecari di Ateneo). In the year 2000, a national inter-university working group, GIM (Gruppo Interuniversitario per il Monitoraggio), formed from representatives from several universities, aimedto develop a University Libraries National Permanent Monitoring Agency. The promotion of the organization of new SBAs, and forming projects of library analysis were some of the most important aims of the workgroup. Between 2000 and 2002 a first nation-wide survey

2 < was conducted by GIM on behalf of CNVSU (Comitato Nazionale per la Valutazione del Sistema Universitario) < A second survey based on the same methodological framework was carried out in 2007 with the help of CRUI (Conferenza dei Rettori delle Università italiane) < The talk will give an overview of the current trends of italian university library systems,through a comparison between the two surveys made by multivariate statistical analysis as well as techniques based on more traditional quantitative measures and indicators. from various different points of view (organization; accessibility; effectiveness and efficiency) There are currently in Italy 88 universities, among them 61 are public and 27 private. 10 upon the 27 private universities are online universities. Within the universities, libraries are more than Students are about and professors more than Librarians are more than

3 At the end of the year 2000 some Italian universities gathered to create the Gruppo Interuniversitario per il Monitoraggio dei sistemi bibliotecari (GIM), that means Interuniversity Group for the monitoring of Italian university libraries. GIM is a non formal group whose aims are to define similar methodologies to measure and evaluate academic library services, to foster a permanent (standing) measurement of academic library services, to encourage the building and further development of library systems within the universities, to develop national and international projects of measurement and evaluation of academic library services. Co-founders of the group were the universities of Bologna, Florence, Milan Bicocca, Padua, Parma, Trento and the Politecnico of Torino. In 2002 GIM planned a project called Measurement and evaluation of Italian academic library services and submitted it to the National Commettee for the Evaluation of Italian Universities (Comitato Nazionale per la Valutazione del Sistema Universitario, CNVSU). The project was approved by the Commettee and partly funded. 3

4 Main objective of the group was to define a set of indicators suitable to be adopted at national level, to allow a benchmarking both between similar libraries within one single university and between different library systems as a whole. The indicators were selected looking at what was already in use in Italian university libraries and at the professional national and international literature. However, also six new indicators were created by the group. The group made a special effort to select and develop a set of indicators that were based on data easy to calculate in the real context of Italian libraries, taking into account the fact that not every library was used to collect a wide range of statistical data. As a result, one set composed by 40 indicators was formed. They were grouped in 5 main areas and meant to allow comparisons between university library systems as a whole and between single similar libraries. 4

5 As a first step the group asked universities to send a list of their libraries. There were two questionnaires: the first for data to be collected at university level, the second for data referred to individual libraries. Both of them were to be filled online. The questions were group into 6 areas that were: Organization of the library system; Facilities and infrastructures; Information provision; Staff; Expenditures; Opening hours, services provided, use of services. 5

6 During 2003 the Group run the first statistical survey. This was the second gathering of statistical data in this field done in Italy after a similar one was done in 1998 with a smaller and slightly different set of indicators. The data collected referred to the previous year (2002). There were respondent libraries upon a total of libraries. A side result of the project was the creation of a complete and controlled list of Italian university libraries. The final report was released and made available online in april 2004 and presented at a national congress held in Padua in In 2007 the group GIM run a second general survey under the patronage of the Conference of Italian University Rectors (CRUI), that is the association of the state and private Italian universities. The survey was done in the second half of the year 2007 and the data gathered referred to the year upon 77 universities answered the questionnaire and libraries were registered. The group decided not to modify the first questionnaire in order to have a series of comparable data and to improve the data themselves. As a result, the survey gave a full picture of Italian academic library services and as a result it became possible for each university to do a benchmarking. Moreover, the questionnaire prepared by the group GIM was adopted and has been used for the last years by a large number of university libraries to collect their own statistical data on a regular basis. 6

7 The data have been analyzed from different points of view. First of all the group tried to identify some trends at national level. Then a benchmarking was made possible through a comparison of the results achieved by the universities. Finally, a multivariate statistical survey was run in order to make it easier a deeper understanding of the results and to identify some trends for different groups of universities, as I am going to show in the following slides. 7

8 These two slides show how the indicators were grouped for the analysis. The indicators were grouped into 5 areas: 1. Indicators that should give an insight to the weight/importance of libraries within the university; 8

9 2. Indicators related to accessibility; 3. Indicators related to efficacy; 4. Indicators related to efficiency; 5. Descriptive indicators. 9

10 The descriptive analysis and the indicators show some quite clear national trends such as the centralization of the University library systems and their razionalisation in terms of more efficiency in spending money. The indicator GIM7 (the number of libraries) has been reduced of about 9% in four years. It seems to confirm the trend already evidenced by the 2002 GIM survey (18% less than in 1998). Looking at the indicator on the Library systems (GIM 30), that collects a series of questions about the systems organization, it seems that the italian universities have been gathering their human and financial resources on their central library divisions. This is also confirmed by the indicators related to the expenditures (GIM28, GIM39) and to the acquisition of the electronic resources (GIM20). Even though the global expenditure of the libraries ( ,00) seems to be the same as the previous GIM survey ( ,00), the indicator that calculates the percentage of the library expenditures on the total university expenditures (GIM28) decreases a little and it is now of about 1,37% which is definetely low compared to the other countries. On the other hand the indicator GIM39 reveals a different distribution of the expenditures between the central divisions and the single libraries: actually there is an increase of about 250% for the central divisions. The expenditure indicators also show us that the more the universities buy electronic resources at a central level, the more they save money for acquisition of new titles. This is even more evident looking at the huge increase of the amount of the e-journals available through the Italian universities (+129%). 10

11 Considering the services offered by the Italian libraries, we can see how they are balanced between the persistency of the traditional library vision and the new perspectives allowed by the digital revolution. The indicators referring to traditional services say that lending is still the core service with about 2,63 loans per user (GIM11 loans/users). It looks like libraries are getting more efficient making about 995 loans for each staff unit as well (GIM23 loans/fte staff). Eventually it is increasing also the percentage of loans on the total library collection (GIM40 loans/collection*100). The stability is also confirmed by the indicator GIM19 (printed journal expenditure/library expenditure*100) which is close to 55%. The library physical spaces have been increasing from the first GIM survey (particularly public spaces) and are now about squared meters. A series of indicators related to the digital libraries could be defined as a sign of innovation. In particular GIM17 (OPAC records/library collection*100), GIM20 (eresources expenditures/library global expenditures*100), GIM38 (e-journals/ejournals+printed journals*100), GIM10 (workstations/seats+worksations*100). The most impressive evidence is the increase of 47% of the OPAC records which means that many universities have moved their old printed catalogues into the OPAC in the last few years. The expenditure for e-resources has doubled reaching about 27 million of euros. About 75% of the library journal collection is electronic. Again the number of workstation on seats has increased even though not as we expected. Maybe in the next future it is going to decrease because more universities are already making available the Wi-fi system to their patrons. 11

12 After calculating indicators and when we tried to interpret the results we obtained comparing the University Library Systems, two questions aroused, concerning common aspects and principally the attempt to improve and make more efficient the comparison analysis between universities: - Is it possible to define some university clusters within which comparisons are so significant as to help us interpreting reality? We can face the problem from two different points of view. We can either choose an a priori classification according to any anagraphic parameter of the universities (such as the enrolment data or the number of faculties) and we analyze the differences of their library performances; or we try a posteriori to cluster universities with similar library systems in order to evaluate if the university structural characteristics are the same. - Is it possible to find a synthesis of indicators, in order to have a smaller set of rankings which make the analysis easier? Also in this case we have more than a single way to reach an answer. We could use an a priori system which weights and aggregates indicators in order to have a single value for each area. Otherwise we could analyze a posteriori the complete data matrix in order to find special syntheses of indicators, to obtain few significant axes where universities can be visually ranked. We tried to answer both questions without referring to a priori choices but using multivariate statistical techniques. We therefore applied cluster analysis to define the presence of university clusters with similar characteristics, and we used factorial analysis to find few factors able to synthesize indicators. 12

13 Clustering o cluster analysis a statistical technique that works on quantitative data means to classify sets of analisys units into previously undefined groups (or clusters), according to characteristics they are defined by. In order to define the similarities between the various sampled units, distances are calculated by using a specifical metrics. We chose Ward clustering algorhythm, which is a hyerarchic agglomeration methods and enables to determine groups with minimal variance and to use euclidean distance, that clusters indicators with similar levels and results. In this case we tried to cluster Universities as statistical units with similar Library Sistems, by using as a parameter the indicators (variables) previously divided according to four different criteria: accessibility, effectiveness, efficiency and structural areas. The method we used did not consider descriptive indicators (considered as bad similarity indicators among universities) as well as universities and indicators with missing values or outliers. Since indicators had different specifical units of measurement, we chose to statistically standardize all data. 13

14 The method we used gave birth to different matrices of distance which created three groups of universities, as the statistical units they contained were clustered: - Cluster 1: Perugia, Pisa University; - Cluster 2: Bergamo, Milano University; - Cluster 3: Sannio, Cassino, University. In order to better understand the clusters we obtained we used a dendrogram. It shows the relations between two elements through the distance between them. By observing the dendrogram we immediately realize three different well delineated clusters. We previously grouped universities by their dimension, using enrolment as a parameter (v. Censis). After the clustering four groups emerged which do not sistematically coincide with the groups we found by means of the cluster analysis. In order to trace the ranking of universities belonging to different dimension groups within the cluster, we highlighted them with different colours representing different dimensions. By analyzing the cluster we immediately understand that very big universities have mostly library system with similar characteristics. Big, middle and small universities, on the contrary, are indistinctly scattered in the three groups. Some considerations: - Milano and Torino Politechnics, although structurally similar and both big universities have different performances. The former is similar to big and very big universities, while the latter is similar to middle and small universities; 14

15 - Most universities with a single library have similar perfomance except of university of Basilicata, with performances similar to middle and big universities, and the university of Molise, with performances similar to big and very big universities; - Small universities, such as la Tuscia, Siena stranieri, Scienze Motorie di Roma, Casamassima have performances similar to big and very big universities such as Padova, Pisa, Palermo, Genova, Verona, Cagliari ; - Private university, such as S. Raffaele di Milano and Castellanza, both small universities with a single library, have also similar perfomances. Factorial analysis is a statistical method aimed to define the fundamental dimensions of the field described by all the variables considered. We must verify the possibility to reach the same descriptive effectiveness with a small set of unobserved variables called factors. This technique was largely used by psychology as mathematical model able to formalize theories about brain and aptitudinal tests and human behaviour. At present this technique is used in different fields such as sociology, psychology and economics. The analysis was performed using the Factor analysis procedures of SPSS 17 sw. 15

16 The analysis showed the presence of two principal factors (absorbing 40% of total variability). In order to analyze them we apply the obsevation of the factorial weight chart, where variables are projected. According to the way they set themselves, we can try to give axes a name. In our case the first axis was called effectiveness axis because it particularly connected to effectiveness indicators, while the second axis can represent the accessibility factor. 16

17 After that, on the axes we extracted and interpreted, we projected the ranking of statistical units. In our case we projected the universities and we evaluated their ranking with respect to an axis representing effectiveness and another representing accessibility. The universities ranked in the upper right quadrant have positive perfomances both for effectiveness and accessibility. On the contrary, universities ranked in the lower left quadrant show the scars levels on both factors. The upper left quadrant shows universities with positive accessibility and negative effectiveness values, while universities in the lower right quadrant have negative accessibility and positive effectiveness values. Each university can trace its ranking of the chart and compare its positiion to other universities. 17

18 Beside enhancing benchmarking, this method can help the performance of a comprehensive data analysis. On the chart it is possible to trace the universities with particular characteristics; by analyzing their setting we can also find possible connections between this charactestic and the factors. This chart shows the setting of a particular type of library system, the single library university, that is the university with a single big library and only a few possible service points. It is interesting to note that these universities have all positive values with respect to the accessibility axis but not to the effectiveness axis. Organizational type then enhances the accessibility of the collection, but the ability to effectively attract users seems to depend upon other variables. 18

19 Along our cluster analysis, we also projected on the factorial axes the university dimension. As in the previous method we can conclude that the dimension is not enough to univocally explain the library system performance, although some significant clusters can be indicative. Middle and big universities are regularly scattered in the quadrants, small universities (which mostly have a single library) are mostly grouped in the positive accessibility semiaxis, while very big universities scarcely have high accessibility values but seem to significantly attract their potential users. 19

20 The work of our team with respect to these methods is just beginning. We still need much time and effort to understand the real survey opportunities offered by these techniques. 20