Reliability and Comparability of Peer Review Results

Transcription

1 Reliability and Comparability of Peer Review Results Nadine Rons, Coordinator of Research Evaluations & Policy Studies Research & Development Department, Vrije Universiteit Brussel Eric Spruyt, Head of the Research Administration Department Universiteit Conference "New Frontiers in Evaluation", Vienna, April 24th-25th 2006.

2 Three cheers for peers Three cheers for peers, Editorial, Nature 439, 118 (12 January 2006). "Thanks are due to researchers who act as referees, as editors resolve their often contradictory advice." "Only in a minority of cases does every referee agree..." Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 2

3 Presentation plan Universiteit I. Validation of results II. III. IV. Reliability & comparability Material investigated 'Ex post' peer review + citation analysis of teams Investigation of results Reliability: inter-peer agreement & different rating habits Comparability: related concepts & intrinsic characteristics Conclusions Aimed at improved results, a better understanding, choosing the right method Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 3

4 I. Validation of results Universiteit 1. Reliability Peer review: principal method to evaluate research quality. BUT: various kinds of bias & different rating habits. & Not always feasible to use measures limiting their influence. Possible to measure reliability? 2. Comparability H F Moed (2005), 'Citation Analysis in Research Evaluation', chapter 18: 'Peer Review and the Validity of Citation Analysis', Springer. More reliable results better correlations with other outcomes? Correlations often relatively weak & depending on the discipline. Can this be explained? (crucial for further acceptance!) Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 4

5 II. Material investigated (Peer review) 1. Peer review Shared principles for the panel-evaluations of teams per discipline: Expertise-based International level Uniform treatment Coherence of results Multi-criteria approach Pertinent advice Exceptions: Different experts for each team (1 discipline at VUB). Specific methodology using different indicators (1 discipline at UA). Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 5

6 II. Material investigated (Peer VUB) VUB-indicators: Standard procedure 'VUB-Richtstramien voor de Disciplinegewijze Onderzoeksevaluaties', VUB Research Council (2001). Scientific merit of the research / uniqueness of the research Research approach / plan / focus / coordination Innovation Quality of the research team Probability that the research objectives will be achieved Research productivity Potential impact on further research and on the development of applications Potential impact for transition to or utility for the community Dominant character of the research (fundamental / applied / policy oriented) Overall research evaluation Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 6

7 II. Material investigated (Peer UA) UA-indicators: 'Protocol 1998' for the Assessment of Research Quality, Association of Universities of the Netherlands (VSNU, 1998). Academic quality Academic productivity Scientific relevance Academic perspective Exception (1 discipline, "partial" indicators): Publications Projects Conference participations Other Globally Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 7

8 II. Material investigated (Citation analysis) 2. Citation analysis 'New Bibliometric Tools for the Assessment of National Research Performance: Database Description, Overview of Indicators and First Apllications', H F Moed et al., Scientometrics 33 (1995). Centre for Science and Technology Studies (CWTS), Leiden University. Thomson ISI citation indexes, corresponding period, same teams. Indicators include: CPP/JCSm: citations / publication with respect to expectations for the journals CPP/FCSm: citations / publication with respect to expectations for the field JCSm/FCSm: journal citation score with respect to expectations for the field Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 8

9 III. Investigation of results (Overview) 1. Reliability a. Inter-peer agreement: Three groups of evaluations according to measured level of agreement. b. Rating habits: Panel-procedures vs. exception with different experts for each team. Influence on results & on correlations between peer review indicators investigated. 2. Comparability a. Related concepts: 'Global' vs. 'partial' indicators & variation with discipline. b. Intrinsic characteristics of methods: Contributions to ratings counted differently & scale effects. Influence on comparability investigated. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 9

10 III. Investigation of results (1. Reliability, a. Inter-peer agreement) 1. Reliability 1. a. Inter-peer agreement In panels: different opinions different positions of teams. Level of inter-peer agreement measured by correlations between the ratings from different peers. 3 groups compared: panels with high, intermediate and low inter-peer agreement. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 10

11 III. Investigation of results (1. Reliability, a. Inter-peer agreement) Influence on results: Results compared to citation analysis: Better inter-peer agreement = higher number of significant correlations, BUT: only at the higher aggregation level of the 3 groups. Other mechanisms have a stronger impact on correlations. Influence on correlations between peer review indicators: Significant correlations for each pair of peer review indicators, for each of the 3 groups (also for indiviual disciplines). Correlations between peer review indicators are relatively robust for variations in inter-peer agreement. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 11

12 III. Investigation of results (1. Reliability, b. Rating habits) 1.b. Rating habits Opinions ratings: according to own habits, reference levels in other evaluations, scores given to other files, known use of scores,... Two cases compared: Exception with different experts for each team scores not necessarily in line with opinions. Standard panel-evaluations uniform reference level. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 12

13 III. Investigation of results (1. Reliability, b. Rating habits) Influence on results: Results compared to citation analysis: Panel-evaluations: significant correlations for all peer review indicators with some or all citation analysis indicators (& vice versa). Different experts: significant correlation for only 1 pair of indicators. Rating habits can influence results significantly. Influence on correlations between peer review indicators: Panel-evaluations: significant correlations for all pairs of indicators. Different experts: significant correlations for only 8% of the pairs. Low observed correlations between indicators (expected to be correlated) can indicate diverging rating habits. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 13

14 III. Investigation of results (2. Comparability, a. Related concepts) 2. Comparability 2.a. Related concepts Partial indicators (publications, projects, conferences,...): no significant correlations between peer review indicators, in contrast to global indicators (scientific merit, productivity, relevance,...). Performances in different activities are not necessarily correlated. Correlations of peer review with citation analysis indicators: the pairs correlating best strongly vary with discipline. An indicator may not represent a same concept for all subject areas. Always use more than one indicator! Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 14

15 III. Investigation of results (2. Comparability, b. Intrinsic characteristics) 2.b. Intrinsic characteristics Contributions to ratings: Different in the minds of peers (pro & contra) and in citation analysis (positive counts). Scale effects: Minimum & maximum limits & their position with respect to the mean value. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 15

16 III. Investigation of results (2. Comparability, b. Intrinsic characteristics) Peer rating frequency distribution: Peer ratings: pro & contra, also elements counted 'negatively'. Scale: minimum & maximum limit. Percentage of the number of teams (58) 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Relative frequency distribution of peer results Scientific merit of the research Uniqueness of the research Research approach / plan / focus / co-ordination Innovation Quality of the research team Probability that the research objectives will be achieved Research productivity Potential impact on further research and on the development of applications Potential for transition to or utility for the community Overall research evaluation Peer results Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 16

17 III. Investigation of results (2. Comparability, b. Intrinsic characteristics) Citation impact frequency distribution: 40% Relative frequency distribution of citation impact All teams in the pure ISI analysis Citation impact: only positive counts, strong influence of highly cited articles. Scale: minimum limit closer to mean & no maximum limit. Percentage of the number of teams (60) 35% 30% 25% 20% 15% 10% 5% CPP/JCSm CPP/FCSm 0% 0,1 0,4 0,7 1 1,3 1,6 1,9 2,2 2,5 2,8 3,1 Indicator value Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 17

18 III. Investigation of results (2. Comparability, b. Intrinsic characteristics) Universiteit Good correlations only when effects of intrinsic characteristics can be filtered out. Field citation impact (CPP/FCSm) 3,0 2,5 2,0 1,5 1,0 Scientific relevance vs. Field citation impact High & intermediate inter-peer agreement group? 0,5 Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 18 0, Peer review "Scientific relevance" score

19 IV. Conclusions Universiteit Reliability Peer review results can be influenced considerably by rating habits. It is recommended to create a uniform reference level (e.g. using panel procedures) or check for signs of low reliability by analysing the outcomes of the peer evaluation itself. Comparability Besides reliability, comparability of results depends on the nature of the indicators, on the subject area, on intrinsic characteristics of the methods,... Different methods describe different aspects. The most suitable method should be carefully chosen or developed. Evaluations should always be based on a series of indicators, never on one single indicator. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 19