Context in performance evaluation

Transcription

1 Context in performance evaluation Dr Lisa Colledge Product Manager, A&G Products, Elsevier, Amsterdam

2 Performance statements without context are meaningless The journal I edit has an Impact Factor of Journal Impact Factor 2009* Pain Nature Genetics Annals of Mathematics Transport This institution s output was 1075 documents last year *Journal Citation Reports 2010 My h-index is 7

3 Coming up in this presentation: the quest for context Our holy grail is a dashboard for every academic that shows their performance in the context of their peer groups and our own internal targets THE QUEST FOR CONTEXT Flexibility in the types of context available is critical Automatic context based on subject field: SNIP

4 Automatic context based on subject field: SNIP Henk F. Moed, Measuring contextual citation impact of scientific journals, Journal of Informetrics, Volume 4, Issue 3, July 2010, Pages

5 Normalisation is very important in journal rankings Context means different things to different people Impact Factor is a good first approximation, but the problem arises of who quotes papers from this journal. It a citation comes from an average or worse source, these citations should be counted lower. Local publications need to be fairly represented Impact Factor evaluation does not take into account the needs of non-english-speaking countries. Must address field differences in citation behaviours If the success of research would only be measured by the Impact Factor, some research areas are handicapped because it is not distributed evenly.

6 Without context in journal ranking Question: Which of these journals ranks highest? Journal Impact Factor 2009* Pain Nature Genetics Annals of Mathematics Transport Progress in Energy and Combustion Science Addiction Biology Remote Sensing of Environment Answer: They are all top of their JCR subject fields *Journal Citation Reports 2010 The context of subject field differs depending on many factors

7 Addressing different citation behaviours between subject fields SNIP: Source-Normalized Impact per Paper SNIP = Raw Impact (average citations) per Paper Citation Potential of journal s field If SNIP >1 then journal ranks above average in its field If SNIP <1 then journal ranks below average in its field

8 Raw Impact (average citations) per Paper Raw Impact per Paper 2010 = Citations received by journal J in 2010 from peer-reviewed content to peer-reviewed content published in Number of peer-reviewed documents published in J in Peer-reviewed content in Scopus = articles, reviews, conference papers

9 Why a 3-year window? To which years are citations made in 2007 directed? Immunology and Microbiology (67,9%) Biochemistry, Genetics and Molecular Biology (65,3%) Pharmacology, Toxicology and Pharmaceutics (63,6%) General (58,3%) Medicine (60,8%) Neuroscience (64,8%) Chemistry (55,4%) Health Professions (56,4%) Nursing (53,5%) Chemical Engineering (51,6%) Materials Science (52,6%) Dentistry (55,9%) Energy (42,1%) Physics and Astronomy (47,3%) Veterinary (47,9%) Agricultural and Biological Sciences (45,0%) Environmental Science (43,5%) Engineering (38,7%) Psychology (44,8%) Earth and Planetary Sciences (41,3%) Computer Science (33,7%) Mathematics (32,5%) Decision Sciences (32,4%) Economics, Econometrics and Finance (29,0%) Business, Management and Accounting (27,7%) Social Sciences (20,1%) Arts and Humanities (10,1%) Source: Felix de Moya

10 Subject field differences addressed in Citation Potential Calculate Citation Potential for Collect papers citing papers in target journal in Collect reference lists of citing papers Etc. 3. Count number of references in citing papers to any (in any journal) 1-3 year old papers 4. Citation Potential = average number of references to any 1-3 year old papers Etc.

11 Subject field is independent of any classification system Journal sets its own subject field i.e. Scopus subject classification is irrelevant Subject field is always up-to-date with changing scope of journal Subject field is always current with increasing coverage of Scopus

12 SNIP accounts for the context of subject field SNIP = Raw Impact (average citations) per Paper Citation Potential of journal s field Life Sciences journal Lots of citations High Raw Impact per Paper High Citation Potential Normalize for differences in citation behaviour between subject fields Arts & Humanities journal Few citations Low Raw Impact per Paper Low Citation Potential

13 Top ranked journals by SNIP Rank Source title SNIP Ca-A Cancer Journal for Clinicians Acta Crystallographica Section A: Foundations of Crystallography Reviews of Modern Physics Vital and health statistics. Series 11, Data from the national health survey Progress in Materials Science National vital statistics reports Physiological Reviews Chemical Reviews Annual Review of Psychology Malayan Nature Journal Annual Review of Immunology Rhinology. Supplement Progress in Energy and Combustion Science Advances in Physics Annual Review of Biochemistry 6.076

14 Screenshot journal analyser and how to get to it

15 Currency, transparency and speed for newly indexed titles Metrics refresh 2010 October 2011 and ongoing April, September Refresh what? Everything recalculated to take into account backwards changes in dynamic database Ensures transparency: metrics relate to current state of Scopus The rule for new titles: if data are available, metric will be calculated If title is covered from 2010, first metrics will be published in 2011 Note first Impact Factor for new launch would be published 2012 Note first Impact Factor for newly indexed existing title would be published 2013

16 Thank you for your attention!