Reliability and Comparability of Peer Review Results

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Reliability and Comparability of Peer Review Results"

Transcription

1 Reliability and Comparability of Peer Review Results Nadine Rons, Coordinator of Research Evaluations & Policy Studies Research & Development Department, Vrije Universiteit Brussel Eric Spruyt, Head of the Research Administration Department Universiteit Conference "New Frontiers in Evaluation", Vienna, April 24th-25th 2006.

2 Three cheers for peers Three cheers for peers, Editorial, Nature 439, 118 (12 January 2006). "Thanks are due to researchers who act as referees, as editors resolve their often contradictory advice." "Only in a minority of cases does every referee agree..." Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 2

3 Presentation plan Universiteit I. Validation of results II. III. IV. Reliability & comparability Material investigated 'Ex post' peer review + citation analysis of teams Investigation of results Reliability: inter-peer agreement & different rating habits Comparability: related concepts & intrinsic characteristics Conclusions Aimed at improved results, a better understanding, choosing the right method Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 3

4 I. Validation of results Universiteit 1. Reliability Peer review: principal method to evaluate research quality. BUT: various kinds of bias & different rating habits. & Not always feasible to use measures limiting their influence. Possible to measure reliability? 2. Comparability H F Moed (2005), 'Citation Analysis in Research Evaluation', chapter 18: 'Peer Review and the Validity of Citation Analysis', Springer. More reliable results better correlations with other outcomes? Correlations often relatively weak & depending on the discipline. Can this be explained? (crucial for further acceptance!) Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 4

5 II. Material investigated (Peer review) 1. Peer review Shared principles for the panel-evaluations of teams per discipline: Expertise-based International level Uniform treatment Coherence of results Multi-criteria approach Pertinent advice Exceptions: Different experts for each team (1 discipline at VUB). Specific methodology using different indicators (1 discipline at UA). Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 5

6 II. Material investigated (Peer VUB) VUB-indicators: Standard procedure 'VUB-Richtstramien voor de Disciplinegewijze Onderzoeksevaluaties', VUB Research Council (2001). Scientific merit of the research / uniqueness of the research Research approach / plan / focus / coordination Innovation Quality of the research team Probability that the research objectives will be achieved Research productivity Potential impact on further research and on the development of applications Potential impact for transition to or utility for the community Dominant character of the research (fundamental / applied / policy oriented) Overall research evaluation Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 6

7 II. Material investigated (Peer UA) UA-indicators: 'Protocol 1998' for the Assessment of Research Quality, Association of Universities of the Netherlands (VSNU, 1998). Academic quality Academic productivity Scientific relevance Academic perspective Exception (1 discipline, "partial" indicators): Publications Projects Conference participations Other Globally Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 7

8 II. Material investigated (Citation analysis) 2. Citation analysis 'New Bibliometric Tools for the Assessment of National Research Performance: Database Description, Overview of Indicators and First Apllications', H F Moed et al., Scientometrics 33 (1995). Centre for Science and Technology Studies (CWTS), Leiden University. Thomson ISI citation indexes, corresponding period, same teams. Indicators include: CPP/JCSm: citations / publication with respect to expectations for the journals CPP/FCSm: citations / publication with respect to expectations for the field JCSm/FCSm: journal citation score with respect to expectations for the field Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 8

9 III. Investigation of results (Overview) 1. Reliability a. Inter-peer agreement: Three groups of evaluations according to measured level of agreement. b. Rating habits: Panel-procedures vs. exception with different experts for each team. Influence on results & on correlations between peer review indicators investigated. 2. Comparability a. Related concepts: 'Global' vs. 'partial' indicators & variation with discipline. b. Intrinsic characteristics of methods: Contributions to ratings counted differently & scale effects. Influence on comparability investigated. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 9

10 III. Investigation of results (1. Reliability, a. Inter-peer agreement) 1. Reliability 1. a. Inter-peer agreement In panels: different opinions different positions of teams. Level of inter-peer agreement measured by correlations between the ratings from different peers. 3 groups compared: panels with high, intermediate and low inter-peer agreement. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 10

11 III. Investigation of results (1. Reliability, a. Inter-peer agreement) Influence on results: Results compared to citation analysis: Better inter-peer agreement = higher number of significant correlations, BUT: only at the higher aggregation level of the 3 groups. Other mechanisms have a stronger impact on correlations. Influence on correlations between peer review indicators: Significant correlations for each pair of peer review indicators, for each of the 3 groups (also for indiviual disciplines). Correlations between peer review indicators are relatively robust for variations in inter-peer agreement. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 11

12 III. Investigation of results (1. Reliability, b. Rating habits) 1.b. Rating habits Opinions ratings: according to own habits, reference levels in other evaluations, scores given to other files, known use of scores,... Two cases compared: Exception with different experts for each team scores not necessarily in line with opinions. Standard panel-evaluations uniform reference level. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 12

13 III. Investigation of results (1. Reliability, b. Rating habits) Influence on results: Results compared to citation analysis: Panel-evaluations: significant correlations for all peer review indicators with some or all citation analysis indicators (& vice versa). Different experts: significant correlation for only 1 pair of indicators. Rating habits can influence results significantly. Influence on correlations between peer review indicators: Panel-evaluations: significant correlations for all pairs of indicators. Different experts: significant correlations for only 8% of the pairs. Low observed correlations between indicators (expected to be correlated) can indicate diverging rating habits. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 13

14 III. Investigation of results (2. Comparability, a. Related concepts) 2. Comparability 2.a. Related concepts Partial indicators (publications, projects, conferences,...): no significant correlations between peer review indicators, in contrast to global indicators (scientific merit, productivity, relevance,...). Performances in different activities are not necessarily correlated. Correlations of peer review with citation analysis indicators: the pairs correlating best strongly vary with discipline. An indicator may not represent a same concept for all subject areas. Always use more than one indicator! Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 14

15 III. Investigation of results (2. Comparability, b. Intrinsic characteristics) 2.b. Intrinsic characteristics Contributions to ratings: Different in the minds of peers (pro & contra) and in citation analysis (positive counts). Scale effects: Minimum & maximum limits & their position with respect to the mean value. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 15

16 III. Investigation of results (2. Comparability, b. Intrinsic characteristics) Peer rating frequency distribution: Peer ratings: pro & contra, also elements counted 'negatively'. Scale: minimum & maximum limit. Percentage of the number of teams (58) 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Relative frequency distribution of peer results Scientific merit of the research Uniqueness of the research Research approach / plan / focus / co-ordination Innovation Quality of the research team Probability that the research objectives will be achieved Research productivity Potential impact on further research and on the development of applications Potential for transition to or utility for the community Overall research evaluation Peer results Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 16

17 III. Investigation of results (2. Comparability, b. Intrinsic characteristics) Citation impact frequency distribution: 40% Relative frequency distribution of citation impact All teams in the pure ISI analysis Citation impact: only positive counts, strong influence of highly cited articles. Scale: minimum limit closer to mean & no maximum limit. Percentage of the number of teams (60) 35% 30% 25% 20% 15% 10% 5% CPP/JCSm CPP/FCSm 0% 0,1 0,4 0,7 1 1,3 1,6 1,9 2,2 2,5 2,8 3,1 Indicator value Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 17

18 III. Investigation of results (2. Comparability, b. Intrinsic characteristics) Universiteit Good correlations only when effects of intrinsic characteristics can be filtered out. Field citation impact (CPP/FCSm) 3,0 2,5 2,0 1,5 1,0 Scientific relevance vs. Field citation impact High & intermediate inter-peer agreement group? 0,5 Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 18 0, Peer review "Scientific relevance" score

19 IV. Conclusions Universiteit Reliability Peer review results can be influenced considerably by rating habits. It is recommended to create a uniform reference level (e.g. using panel procedures) or check for signs of low reliability by analysing the outcomes of the peer evaluation itself. Comparability Besides reliability, comparability of results depends on the nature of the indicators, on the subject area, on intrinsic characteristics of the methods,... Different methods describe different aspects. The most suitable method should be carefully chosen or developed. Evaluations should always be based on a series of indicators, never on one single indicator. Nadine Rons (Vrije Universiteit Brussel) & Eric Spruyt (Universiteit ) pag. 19

Impact measures of interdisciplinary research in physics

Impact measures of interdisciplinary research in physics Jointly published by Akadémiai Kiadó, Budapest Scientometrics, and Kluwer Academic Publishers, Dordrecht Vol. 53, No. 2 (2002) 241 248 Impact measures of interdisciplinary research in physics ED J. RINIA,

More information

Bibliometric study on Dutch Open Access published output 2000-2012/2013

Bibliometric study on Dutch Open Access published output 2000-2012/2013 Bibliometric study on Dutch Open Access published output 2000-2012/2013 Preliminary research report to the Dutch Ministry of Science, Culture & Education (OC&W) January 2015 Center for Science & Technology

More information

Research potential of Madurai Kamaraj University A Scientometric view

Research potential of Madurai Kamaraj University A Scientometric view Original Article International Journal of Educational Research and Technology Volume 2, Issue 2, December 2011: 62-66 ISSN 0976-4089 Journal s URL: www.soeagra.com/ijert.htm Research potential of Madurai

More information

Measurement of Central Aspects of Scientific Research: Performance, Interdisciplinarity, Structure

Measurement of Central Aspects of Scientific Research: Performance, Interdisciplinarity, Structure MEASUREMENT, 3(1), 1 19 Copyright 2005, Lawrence Erlbaum Associates, Inc. FOCUS ARTICLE Measurement of Central Aspects of Scientific Research: Performance, Interdisciplinarity, Structure Anthony F. J.

More information

Journal of Informetrics

Journal of Informetrics Journal of Informetrics 5 (20) 8 86 Contents lists available at ScienceDirect Journal of Informetrics journal homepage: www.elsevier.com/locate/joi A proposal for a First-Citation-Speed-Index L. Egghe

More information

Challenges in Ranking of Universities

Challenges in Ranking of Universities Challenges in Ranking of Universities Invited paper for the First International Conference on World Class Universities, Shanghai Jaio Tong University, Shanghai, June 16-18, 2005 Abstract Anthony F.J. van

More information

A Bibliometric Comparison of the Research of Three UK Business Schools

A Bibliometric Comparison of the Research of Three UK Business Schools , March 12-14, 2014, Hong Kong A Bibliometric Comparison of the Research of Three UK Business Schools John C. Mingers, Evangelia A. Lipitakis Abstract The Leiden ranking methodology (LRM), also sometimes

More information

Publishing in Nature journals

Publishing in Nature journals Publishing in Nature journals John Plummer Senior Editor Nature Materials November 2015 By the end of this talk I hope you will: Appreciate the key editorial differences between Nature journals. Better

More information

Guide for Authors MORAVIAN GEOGRAPHICAL REPORTS

Guide for Authors MORAVIAN GEOGRAPHICAL REPORTS MORAVIAN GEOGRAPHICAL REPORTS Guide for Authors Introduction Moravian Geographical Reports [MGR] is an international peer-reviewed journal, which has been published in English continuously since 1993 by

More information

Scientometrics as Big Data Science:

Scientometrics as Big Data Science: Scientometrics as Big Data Science: On integration of data sources and the problem of different types of classification systems Henk F. Moed (Elsevier, The Netherlands) With: Marc Luwel (Hercules Foundation,

More information

Journal Impact Factor, Eigenfactor, Journal Influence and Article Influence

Journal Impact Factor, Eigenfactor, Journal Influence and Article Influence DEPARTMENT OF ECONOMICS AND FINANCE COLLEGE OF BUSINESS AND ECONOMICS UNIVERSITY OF CANTERBURY CHRISTCHURCH, NEW ZEALAND Journal Impact Factor, Eigenfactor, Journal Influence and Article Influence Chia-Lin

More information

Q+A Memo BACKGROUND. Why is U-Multirank an important development in higher education?

Q+A Memo BACKGROUND. Why is U-Multirank an important development in higher education? Q+A Memo BACKGROUND Why is U-Multirank an important development in higher education? Europe (and the world) needs a wide diversity of higher education institutions which do well in different areas, to

More information

Urban Andersson Jonas Gilbert and Karin Henning Gothenburg University Library Gothenburg, Sweden

Urban Andersson Jonas Gilbert and Karin Henning Gothenburg University Library Gothenburg, Sweden Date submitted: 03/07/2010 Download data versus traditional impact metrics: Measuring impact in a sample of biomedical doctoral dissertations Urban Andersson Jonas Gilbert and Karin Henning Gothenburg

More information

The journal coverage of bibliometric databases: A comparison of Scopus and Web of Science

The journal coverage of bibliometric databases: A comparison of Scopus and Web of Science The journal coverage of bibliometric databases: A comparison of and Web of Science Philippe Mongeon and Adèle Paul-Hus École de bibliothéconomie et des sciences de l information, Université de Montréal

More information

Challenges in Ranking of Universities

Challenges in Ranking of Universities Challenges in Ranking of Universities First International Conference on World Class Universities, Jaio Tong University, Shanghai, June 16-18, 2005 Anthony F.J. van Raan Center for Science and technology

More information

Department of Biological Sciences, National University of Singapore, Singapore

Department of Biological Sciences, National University of Singapore, Singapore 1 2 3 4 5 6 Extreme inequalities of citation counts in environmental sciences Deepthi Chimalakonda 1, Alex R. Cook 2, 3, L. Roman Carrasco 1,* 1 Department of Biological Sciences, National University of

More information

Europass Curriculum Vitae

Europass Curriculum Vitae Europass Curriculum Vitae Personal information First name(s) / Surname(s) E-mail Flavia Di Costa dicosta@dis.uniroma1.it; Work experience Dates December 2013 Onwards Research fellow Research activity on

More information

RANKING THE SCHOLARLY IMPACT OF NEW JERSEY COLLEGES AND UNIVERSITIES IN THE FIELD OF PSYCHOLOGY

RANKING THE SCHOLARLY IMPACT OF NEW JERSEY COLLEGES AND UNIVERSITIES IN THE FIELD OF PSYCHOLOGY TCNJ JOURNAL OF STUDENT SCHOLARSHIP VOLUME XII APRIL, 2010 RANKING THE SCHOLARLY IMPACT OF NEW JERSEY COLLEGES AND UNIVERSITIES IN THE FIELD OF PSYCHOLOGY Author: Michelle T. Guerin Faculty Sponsor: John

More information

Bibliometric analysis tools on top of the university s bibliographic database, new roles and opportunities for library outreach

Bibliometric analysis tools on top of the university s bibliographic database, new roles and opportunities for library outreach Bibliometric analysis tools on top of the university s bibliographic database, new roles and opportunities for library outreach Österreichische Bibliothekartag W. Gerritsma & P.L. van der Togt Wageningen

More information

and Funding. to show that its direct costs are smaller in proportion to the funding allocated than those of the Research Councils, for example.

and Funding. to show that its direct costs are smaller in proportion to the funding allocated than those of the Research Councils, for example. University of Warwick Department of Economics QR and Metrics Prepared by Mark Harrison and Michael Waterson This version 30 June 2006; first draft 26 June 2006. Summary. This note presents strong arguments

More information

Metrics on the rise their increasing role in research assessment

Metrics on the rise their increasing role in research assessment Metrics on the rise their increasing role in research assessment 1 Elsevier Research Trends Conference Big-Data & E-Science Canberra 15-16 May 2012 Linda Butler Australian National University and University

More information

Policy on Academic Tracks and Promotions for the School of Nursing (SON) at the American University of Beirut (AUB)

Policy on Academic Tracks and Promotions for the School of Nursing (SON) at the American University of Beirut (AUB) Policy on Academic Tracks and Promotions for the School of Nursing (SON) at the American University of Beirut (AUB) Preamble The School of Nursing currently has 2 tracks, Academic and Clinical. The Academic

More information

THE USE OF THOMSON REUTERS RESEARCH ANALYTIC RESOURCES IN ACADEMIC PERFORMANCE EVALUATION DR. EVANGELIA A.E.C. LIPITAKIS OCTOBER 2014, PRAGUE

THE USE OF THOMSON REUTERS RESEARCH ANALYTIC RESOURCES IN ACADEMIC PERFORMANCE EVALUATION DR. EVANGELIA A.E.C. LIPITAKIS OCTOBER 2014, PRAGUE THE USE OF THOMSON REUTERS RESEARCH ANALYTIC RESOURCES IN ACADEMIC PERFORMANCE EVALUATION DR. EVANGELIA A.E.C. LIPITAKIS OCTOBER 2014, PRAGUE Thomson Reuters: Solutions Portfolio to Seamlessly Support

More information

Journal Ranking & Impact Factors

Journal Ranking & Impact Factors [Type here] Journal Ranking & Impact Factors This practical workshop aims to highlight the main tools available to identify and compare journal rankings and impact factors. Their relative strengths and

More information

What Do Experts Know About Forecasting Journal Quality? A Comparison with ISI Research Impact in Finance*

What Do Experts Know About Forecasting Journal Quality? A Comparison with ISI Research Impact in Finance* What Do Experts Know About Forecasting Journal Quality? A Comparison with ISI Research Impact in Finance* Chia-Lin Chang Department of Applied Economics Department of Finance National Chung Hsing University

More information

Correlation key concepts:

Correlation key concepts: CORRELATION Correlation key concepts: Types of correlation Methods of studying correlation a) Scatter diagram b) Karl pearson s coefficient of correlation c) Spearman s Rank correlation coefficient d)

More information

Solving Three-objective Optimization Problems Using Evolutionary Dynamic Weighted Aggregation: Results and Analysis

Solving Three-objective Optimization Problems Using Evolutionary Dynamic Weighted Aggregation: Results and Analysis Solving Three-objective Optimization Problems Using Evolutionary Dynamic Weighted Aggregation: Results and Analysis Abstract. In this paper, evolutionary dynamic weighted aggregation methods are generalized

More information

Pegasus Guidelines for Applicants

Pegasus Guidelines for Applicants Pegasus Guidelines for Applicants 1 These guidelines are valid for both Pegasus-long fellowships and Pegasus-short fellowships. 1 There are 6 calls in total, distributed over 3 years: Call deadline Category

More information

Scientific research competitiveness of world universities in computer science

Scientific research competitiveness of world universities in computer science Jointly published by Akadémiai Kiadó, Budapest Scientometrics, Vol. 76, No. 2 (2008) 245 260 and Springer, Dordrecht DOI: 10.1007/s11192-007-1913-7 Scientific research competitiveness of world universities

More information

CITATION RESOURCES RESEARCH SKILLS. Definitions of terms

CITATION RESOURCES RESEARCH SKILLS. Definitions of terms CITATION RESOURCES Citation analysis assumes that research published in prestigious journals with high impact scores and research published by scientists who enjoy high impact ratings will be cited more

More information

Journals in Developmental Psychology

Journals in Developmental Psychology Journals in Developmental Psychology Developmental Journals What makes a journal developmental? What makes your research developmental? Examples of appropriate content as defined by the journal Child Development

More information

Do Supplemental Online Recorded Lectures Help Students Learn Microeconomics?*

Do Supplemental Online Recorded Lectures Help Students Learn Microeconomics?* Do Supplemental Online Recorded Lectures Help Students Learn Microeconomics?* Jennjou Chen and Tsui-Fang Lin Abstract With the increasing popularity of information technology in higher education, it has

More information

Net Impact Factor and Immediacy Index of ten frontline ISI journals: A case study based on citations to in-press or unpublished articles

Net Impact Factor and Immediacy Index of ten frontline ISI journals: A case study based on citations to in-press or unpublished articles Net Impact Factor and Immediacy Index of ten frontline ISI journals: A case study based on citations to in-press or unpublished articles E. R. Prakasan*; Anil Sagar; V. L. Kalyane; and Vijai Kumar Scientific

More information

III Seminario EC3. Rankings ISI de universidades españolas por campos científicos

III Seminario EC3. Rankings ISI de universidades españolas por campos científicos III Seminario EC3 sobre evaluación de la ciencia y la comunicación científica Rankings ISI de universidades españolas por campos científicos Francisco Herrera Dpto. Ciencias de la Computación e Inteligencia

More information

Impact of Research. John Peters Joint CEO Emerald Global, Bingley UK President Emerald Group Publishing Inc, Cambridge Mass.

Impact of Research. John Peters Joint CEO Emerald Global, Bingley UK President Emerald Group Publishing Inc, Cambridge Mass. Impact of Research John Peters Joint CEO Emerald Global, Bingley UK President Emerald Group Publishing Inc, Cambridge Mass. March 2010 Aim of this presentation Share experience and thoughts on research

More information

English Summary 1. cognitively-loaded test and a non-cognitive test, the latter often comprised of the five-factor model of

English Summary 1. cognitively-loaded test and a non-cognitive test, the latter often comprised of the five-factor model of English Summary 1 Both cognitive and non-cognitive predictors are important with regard to predicting performance. Testing to select students in higher education or personnel in organizations is often

More information

PPRN Meeting Norfolk 2014. Prof. dr. Michaël Dooms Michael.Dooms@vub.ac.be

PPRN Meeting Norfolk 2014. Prof. dr. Michaël Dooms Michael.Dooms@vub.ac.be Integrating triple P bottom line performance and local community perception on the license to operate : towards new partnerships between port cluster stakeholders PPRN Meeting Norfolk 2014 Prof. dr. Michaël

More information

A DEFICIENCY IN THE ALGORITHM FOR CALCULATING THE IMPACT FACTOR OF SCHOLARLY JOURNALS: THE JOURNAL IMPACT FACTOR

A DEFICIENCY IN THE ALGORITHM FOR CALCULATING THE IMPACT FACTOR OF SCHOLARLY JOURNALS: THE JOURNAL IMPACT FACTOR A DEFICIENCY IN THE ALGORITHM FOR CALCULATING THE IMPACT FACTOR OF SCHOLARLY JOURNALS: THE JOURNAL IMPACT FACTOR Péter Jacsó (Department of Information and Computer Science, University of Hawai i at Mãnoa,

More information

Hirsch s index: a case study conducted at the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, Universidade de São Paulo

Hirsch s index: a case study conducted at the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, Universidade de São Paulo Brazilian Journal of Medical and Biological Research (2007) 40: 1529-1536 Hirsch s index: a case study conducted at the FFCLRP, USP ISSN 0100-879X 1529 Hirsch s index: a case study conducted at the Faculdade

More information

Comparing two thermometers : Impact factors of 20 leading economic journals according to Journal Citation Reports and Scopus

Comparing two thermometers : Impact factors of 20 leading economic journals according to Journal Citation Reports and Scopus Comparing two thermometers : Impact factors of 20 leading economic journals according to Journal Citation Reports and Scopus VLADIMIR PISLYAKOV Higher School of Economics, Library, Moscow (Russia) Impact

More information

Publication Bias in medical research: issues and communities

Publication Bias in medical research: issues and communities Publication Bias in medical research: issues and communities Edgar Schiebel and Maria Elisabeth Züger edgar.schiebel@ait.ac.at, maria-elisabeth.zueger@ait.ac.at AIT Austrian Institute of Technology GmbH,

More information

Mining Google Scholar Citations: An Exploratory Study

Mining Google Scholar Citations: An Exploratory Study Mining Google Scholar Citations: An Exploratory Study Ze Huang and Bo Yuan Intelligent Computing Lab, Division of Informatics, Graduate School at Shenzhen, Tsinghua University, Shenzhen 518055, P.R. China

More information

Purpose and principles of review

Purpose and principles of review http.//www.eua.be 1 Purpose and principles of review Addresses the most popular global university rankings Providing universities with analysis of the methodologies Only publicly accessible information

More information

Guidelines for Preparation of Review Protocols

Guidelines for Preparation of Review Protocols Guidelines for Preparation of Review Protocols Type of document: Policy _x_ Guideline Procedure Version: 1.0, January 1 2001 Decision: Steering Group, Date? A Campbell Systematic Review is meant to review

More information

NORTHERN VIRGINIA COMMUNITY COLLEGE PSYCHOLOGY 211 - RESEARCH METHODOLOGY FOR THE BEHAVIORAL SCIENCES Dr. Rosalyn M.

NORTHERN VIRGINIA COMMUNITY COLLEGE PSYCHOLOGY 211 - RESEARCH METHODOLOGY FOR THE BEHAVIORAL SCIENCES Dr. Rosalyn M. NORTHERN VIRGINIA COMMUNITY COLLEGE PSYCHOLOGY 211 - RESEARCH METHODOLOGY FOR THE BEHAVIORAL SCIENCES Dr. Rosalyn M. King, Professor DETAILED TOPICAL OVERVIEW AND WORKING SYLLABUS CLASS 1: INTRODUCTIONS

More information

Memorandum of Understanding for Financial Co Sponsorship of the IEEE Transactions on Cloud Computing June 2012

Memorandum of Understanding for Financial Co Sponsorship of the IEEE Transactions on Cloud Computing June 2012 Memorandum of Understanding for Financial Co Sponsorship of the IEEE Transactions on Cloud Computing June 2012 Introduction This document serves as an agreement between the IEEE Computer Society, the IEEE

More information

FINDING MEANINGFUL PERFORMANCE MEASURES FOR HIGHER EDUCATION A REPORT FOR EXECUTIVES

FINDING MEANINGFUL PERFORMANCE MEASURES FOR HIGHER EDUCATION A REPORT FOR EXECUTIVES FINDING MEANINGFUL PERFORMANCE MEASURES FOR HIGHER EDUCATION A REPORT FOR EXECUTIVES .......................................................................................................................

More information

The impact factor s Matthew effect: a natural experiment in bibliometrics

The impact factor s Matthew effect: a natural experiment in bibliometrics The impact factor s Matthew effect: a natural experiment in bibliometrics Vincent Larivière and Yves Gingras Observatoire des sciences et des technologies (OST) Centre interuniversitaire de recherche sur

More information

Social Media Mining. Network Measures

Social Media Mining. Network Measures Klout Measures and Metrics 22 Why Do We Need Measures? Who are the central figures (influential individuals) in the network? What interaction patterns are common in friends? Who are the like-minded users

More information

Objectives 1. Identify your potential contributions to the expansion of nursing knowledge/evidence. 2. Review essential writing habits.

Objectives 1. Identify your potential contributions to the expansion of nursing knowledge/evidence. 2. Review essential writing habits. Getting Published in Rehab Nursing September 9, 2014 Elaine T. Miller, PhD, RN, CRRN, FAAN FAHA Editor, Rehabilitation Nursing Objectives 1. Identify your potential contributions to the expansion of nursing

More information

Jörg Neufeld, Nathalie Huber and Antje Wegner

Jörg Neufeld, Nathalie Huber and Antje Wegner On the Relationship between Peer Review-based Selection Decisions in Individual Research Funding and Applicants Publication Performance: The Case of the ERC Starting Grants 1 Jörg Neufeld, Nathalie Huber

More information

BRING YOUR RESEARCH TO LIFE

BRING YOUR RESEARCH TO LIFE NIHES AT A GLANCE RESEARCH TRAINING IN MEDICINE AND THE HEALTH SCIENCES BRING YOUR RESEARCH TO LIFE 71% of all NIHES papers are published in the top 10% of the highest-impact journals. Source: Journal

More information

Working Paper Do rankings reflect research quality?

Working Paper Do rankings reflect research quality? econstor www.econstor.eu Der Open-Access-Publikationsserver der ZBW Leibniz-Informationszentrum Wirtschaft The Open Access Publication Server of the ZBW Leibniz Information Centre for Economics Frey, Bruno

More information

Evaluation of research impact: the Dutch experience

Evaluation of research impact: the Dutch experience Evaluation of research impact: the Dutch experience Robert Tijssen Center for Science and Technology Studies (CWTS) Leiden University The Netherlands Presentation at UQ/UWA/TR Conference National and international

More information

Ethical Standards for Publication of Aeronautics and Astronautics Research

Ethical Standards for Publication of Aeronautics and Astronautics Research Ethical Standards for Publication of Aeronautics and Astronautics Research I. Preface/Introduction II. AIAA Ethical Standards A. Ethical Standards for AIAA Journals B. Ethical Standards for AIAA Books

More information

Analyzing Research Articles: A Guide for Readers and Writers 1. Sam Mathews, Ph.D. Department of Psychology The University of West Florida

Analyzing Research Articles: A Guide for Readers and Writers 1. Sam Mathews, Ph.D. Department of Psychology The University of West Florida Analyzing Research Articles: A Guide for Readers and Writers 1 Sam Mathews, Ph.D. Department of Psychology The University of West Florida The critical reader of a research report expects the writer to

More information

EVALUATION GUIDE Additional Information. FCT Evaluation of R&D Units 2013

EVALUATION GUIDE Additional Information. FCT Evaluation of R&D Units 2013 EVALUATION GUIDE Additional Information FCT Evaluation of R&D Units 2013 29 April 2014 FCT Evaluation of R&D Units 2013 Additional Information to the Evaluation Guide The following document is an update

More information

1. FORMULATING A RESEARCH QUESTION

1. FORMULATING A RESEARCH QUESTION W R I T I N G A R E S E A R C H G R A N T P R O P O S A L 1. FORMULATING A RESEARCH QUESTION 1.1. Identify a broad area of interest through literature searches, discussions with colleagues, policy makers

More information

Bibliometric Big Data and its Uses. Dr. Gali Halevi Elsevier, NY

Bibliometric Big Data and its Uses. Dr. Gali Halevi Elsevier, NY Bibliometric Big Data and its Uses Dr. Gali Halevi Elsevier, NY In memoriam https://www.youtube.com/watch?v=srbqtqtmncw The Multidimensional Research Assessment Matrix Unit of assessment Purpose Output

More information

RESEARCH ASSESSMENT IN ROMANIA. Răzvan Valentin Florian

RESEARCH ASSESSMENT IN ROMANIA. Răzvan Valentin Florian RESEARCH ASSESSMENT IN ROMANIA Răzvan Valentin Florian CONTEXT 2002: Ad Astra association founded 2007: Presidential Commission for the analysis and development of Romanian science and education policies

More information

The challenge of writing a PhD proposal in honours (undergraduate) education;

The challenge of writing a PhD proposal in honours (undergraduate) education; The challenge of writing a PhD proposal in honours (undergraduate) education; Fred Wiegant*, Karin Scager, Ton Peeters and Johannes Boonstra Groningen 4& 5 october 2012 *Fred Wiegant Institute of Education,

More information

POLICY AND PROCEDURES FOR MEASUREMENT OF RESEARCH OUTPUT OF PUBLIC HIGHER EDUCATION INSTITUTIONS MINISTRY OF EDUCATION

POLICY AND PROCEDURES FOR MEASUREMENT OF RESEARCH OUTPUT OF PUBLIC HIGHER EDUCATION INSTITUTIONS MINISTRY OF EDUCATION POLICY AND PROCEDURES FOR MEASUREMENT OF RESEARCH OUTPUT OF PUBLIC HIGHER EDUCATION INSTITUTIONS MINISTRY OF EDUCATION June 2003 TABLE OF CONTENTS 1. Introduction...3 2. Policy Framework...3 3. Recognised

More information

BACHELOR OF BUSINESS ADMINISTRATION Program: Goals and Objectives

BACHELOR OF BUSINESS ADMINISTRATION Program: Goals and Objectives BACHELOR OF BUSINESS ADMINISTRATION Program: Goals and Objectives 1. A Seidman BBA graduate will be an effective communicator. He/she will be able to: 1.1 Engage in effective interpersonal dialogue. 1.2

More information

Department of Marketing Promotion and Tenure Guidelines February 2011

Department of Marketing Promotion and Tenure Guidelines February 2011 Department of Marketing Promotion and Tenure Guidelines February 2011 I. Overview This document describes expectations for all tenured and tenure track faculty in the Department of Marketing to supplement

More information

Short title: Empirical evaluation of Internet indexes. Please send correspondence to both authors:

Short title: Empirical evaluation of Internet indexes. Please send correspondence to both authors: Short title: Empirical evaluation of Internet indexes Please send correspondence to both authors: Wouter Mettrop 1 and Paul Nieuwenhuysen 2 1. CWI - Centrum voor Wiskunde en Informatica / Library, Kruislaan

More information

Linking Human Resources in Science and Technology and Innovative Performance

Linking Human Resources in Science and Technology and Innovative Performance Linking Human Resources in Science and Technology and Innovative Performance Developing Concordance Schemes and Indicators to Enable the Analysis of Education - Science Technology Industry Relationships

More information

DEPARTMENT OF MARKETING COLLEGE OF BUSINESS ADMINSTRATION POLICY ON REAPPOINTMENT, TENURE, AND PROMOTION (RTP)

DEPARTMENT OF MARKETING COLLEGE OF BUSINESS ADMINSTRATION POLICY ON REAPPOINTMENT, TENURE, AND PROMOTION (RTP) Approved by Academic Affairs May 2010 DEPARTMENT OF MARKETING COLLEGE OF BUSINESS ADMINSTRATION POLICY ON REAPPOINTMENT, TENURE, AND PROMOTION (RTP) I. DEPARTMENT OF MARKETING RTP POLICY A. Preamble B.

More information

There is only one compliance check required. See the answer to Question # 5 above.

There is only one compliance check required. See the answer to Question # 5 above. A Request for Proposals for Merit Peer Review Services for Scientific and Education Research Applications RFP #15666 Questions and Answers July 29 8/26/14 1. The Staffing and Project Management section

More information

InBev-BAILLET LATOUR GRANTS FOR MEDICAL RESEARCH 2016 : Infectious Diseases APPLICATION FORM

InBev-BAILLET LATOUR GRANTS FOR MEDICAL RESEARCH 2016 : Infectious Diseases APPLICATION FORM InBev-BAILLET LATOUR GRANTS FOR MEDICAL RESEARCH 2016 : Infectious Diseases APPLICATION FORM This file must be filled in electronically in English (font Century Gothic, font size 10, not exceeding the

More information

Types of Group Comparison Research. Stephen E. Brock, Ph.D., NCSP EDS 250. Causal-Comparative Research 1

Types of Group Comparison Research. Stephen E. Brock, Ph.D., NCSP EDS 250. Causal-Comparative Research 1 Causal-Comparative Research & Single Subject Research Stephen E. Brock, Ph.D., NCSP California State University, Sacramento 1 Correlation vs. Group Comparison Correlational Group Comparison 1 group 2 or

More information

Psychology Department Faculty Evaluation Form for Years 2008-2010

Psychology Department Faculty Evaluation Form for Years 2008-2010 Psychology Department Faculty Evaluation Form for Years 2008-2010 In 2009, the Provost s Office directed Deans and Heads to assure that faculty evaluations take into consideration efforts to promote University

More information

CITATION METRICS WORKSHOP ANALYSIS & INTERPRETATION WEB OF SCIENCE Prepared by Bibliometric Team, NUS Libraries. April 2014.

CITATION METRICS WORKSHOP ANALYSIS & INTERPRETATION WEB OF SCIENCE Prepared by Bibliometric Team, NUS Libraries. April 2014. CITATION METRICS WORKSHOP ANALYSIS & INTERPRETATION WEB OF SCIENCE Prepared by Bibliometric Team, NUS Libraries. April 2014. Analysis & Interpretation of Results using Web of Science Steps Technique Page

More information

On becoming a high impact journal in international business and management

On becoming a high impact journal in international business and management On becoming a high impact journal in international business and management Anne-Wil Harzing Version January 2008 Accepted for European Journal of International Management, vol. 2/2 Copyright 2008 Anne-Wil

More information

The Dutch Experience in Quality Assurance

The Dutch Experience in Quality Assurance The Dutch Experience in Quality Assurance La Qualità nello Spazio Europeo dell Istuzione Ascoli Piceno 4-55 November 2005 Robert Wagenaar (University of Groningen, NL) The Dutch Higher Education System:

More information

The ACGME Resident Survey Aggregate Reports: An Analysis and Assessment of Overall Program Compliance

The ACGME Resident Survey Aggregate Reports: An Analysis and Assessment of Overall Program Compliance The ACGME Resident Survey Aggregate Reports: An Analysis and Assessment of Overall Program Compliance Kathleen D. Holt, PhD Rebecca S. Miller, MS Abstract Background The Accreditation Council for Graduate

More information

Chapter 2 Measuring the Impact of Articles, Journals and Nations

Chapter 2 Measuring the Impact of Articles, Journals and Nations Chapter 2 Measuring the Impact of Articles, Journals and Nations Abstract The rise of metrics that measure scientific production and impact is increasingly important. The system of determining impact of

More information

Marketing Journal and Paper Impact: Some Additional Evidence. Geoffrey N. Soutar. University of Western Australia Business School.

Marketing Journal and Paper Impact: Some Additional Evidence. Geoffrey N. Soutar. University of Western Australia Business School. Marketing Journal and Paper Impact: Some Additional Evidence Geoffrey N. Soutar University of Western Australia Business School Abstract The present study updated Soutar and Murphy s (2009) study of the

More information

Methods for Firewall Policy Detection and Prevention

Methods for Firewall Policy Detection and Prevention Methods for Firewall Policy Detection and Prevention Hemkumar D Asst Professor Dept. of Computer science and Engineering Sharda University, Greater Noida NCR Mohit Chugh B.tech (Information Technology)

More information

Science in Transition An Agenda for Change version 16 Oct. 2013

Science in Transition An Agenda for Change version 16 Oct. 2013 Science in Transition An Agenda for Change version 16 Oct. 2013 Science is broadly appreciated. And rightfully so. Its successes are unfathomable, and modern societies are inconceivable without its influence.

More information

Does it Matter Which Citation Tool is Used to Compare the h-index of a Group of Highly Cited Researchers?

Does it Matter Which Citation Tool is Used to Compare the h-index of a Group of Highly Cited Researchers? Australian Journal of Basic and Applied Sciences, 7(4): 198-202, 2013 ISSN 1991-8178 Does it Matter Which Citation Tool is Used to Compare the h-index of a Group of Highly Cited Researchers? 1 Hadi Farhadi,

More information

Challenges in Cross-National Comparisons of R&D Expenditure and Publication Output

Challenges in Cross-National Comparisons of R&D Expenditure and Publication Output Challenges in Cross-National Comparisons of R&D Expenditure and Publication Output Kaja Wendt, * Dag W. Aksnes, * Gunnar Sivertsen * and Staffan Karlsson ** * Kaja.Wendt@nifu.no; Dag.W.Aksnes@nifu.no;

More information

EXAMPLE OF AN APPLICATION FORM: OUTGOING [PEGASUS]² Marie Skłodowska- Curie Fellowships

EXAMPLE OF AN APPLICATION FORM: OUTGOING [PEGASUS]² Marie Skłodowska- Curie Fellowships EXAMPLE OF AN APPLICATION FORM: OUTGOING [PEGASUS]² Marie Skłodowska- Curie Fellowships GENERAL HOST INSTITUTION HEAD OF RESEARCH UNIT EXTRA DATA ETHICS EXPERT PANEL REFEREES PERSONAL DATA PROJECT OUTLINE

More information

Chapter 5. Phrase-based models. Statistical Machine Translation

Chapter 5. Phrase-based models. Statistical Machine Translation Chapter 5 Phrase-based models Statistical Machine Translation Motivation Word-Based Models translate words as atomic units Phrase-Based Models translate phrases as atomic units Advantages: many-to-many

More information

Reviewer Information THE IMPORTANCE OF PEER REVIEW

Reviewer Information THE IMPORTANCE OF PEER REVIEW American Journal of Preventive Medicine Sponsored by the American College of Preventive Medicine and the Association for Prevention Teaching and Research Reviewer Information The American Journal of Preventive

More information

Information and Employee Evaluation: Evidence from a Randomized Intervention in Public Schools. Jonah E. Rockoff 1 Columbia Business School

Information and Employee Evaluation: Evidence from a Randomized Intervention in Public Schools. Jonah E. Rockoff 1 Columbia Business School Preliminary Draft, Please do not cite or circulate without authors permission Information and Employee Evaluation: Evidence from a Randomized Intervention in Public Schools Jonah E. Rockoff 1 Columbia

More information

Clemson University College of Health, Education, and Human Development School of Nursing Promotion, Tenure, and Appointment Renewal Guidelines

Clemson University College of Health, Education, and Human Development School of Nursing Promotion, Tenure, and Appointment Renewal Guidelines Approved SON Faculty September 2000 1 Clemson University College of Health, Education, and Human Development School of Nursing Promotion, Tenure, and Appointment Renewal Guidelines The School of Nursing

More information

Report to the 79 th Legislature. Use of Credit Information by Insurers in Texas

Report to the 79 th Legislature. Use of Credit Information by Insurers in Texas Report to the 79 th Legislature Use of Credit Information by Insurers in Texas Texas Department of Insurance December 30, 2004 TABLE OF CONTENTS Executive Summary Page 3 Discussion Introduction Page 6

More information

Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods

Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods Dag W. Aksnes 1, Jesper W. Schneider 2, Magnus Gunnarsson 3 1 dag.w.aksnes@nifu.no

More information

The Journals of Gerontology, Series B: Psychological Sciences and Social Sciences

The Journals of Gerontology, Series B: Psychological Sciences and Social Sciences The Journals of Gerontology, Series B: Psychological Sciences and Social Sciences The Journals of Gerontology: Series B is a bimonthly journal of The Gerontological Society of America, The Social Sciences

More information

Department of Business, Management and Accounting

Department of Business, Management and Accounting Department of Business, Management and Accounting Departmental Promotion and Tenure Criteria March 22, 2006 I. Introduction The quality of programs offered in the Department of Business, Management and

More information

An Overview. University. Performance Metrics. and. Rankings

An Overview. University. Performance Metrics. and. Rankings An Overview of University Performance Metrics and Rankings by Dr. Dario Toncich (PhD, MEng, BEng(Elec)(Hons) 1. Introduction Universities around the world are regularly assessed in terms of their performance

More information

Eclipse User Guide. 2010 Candidate, Vacancy and Client Contact Matching. Candidate, Vacancy and Client Contact Matching

Eclipse User Guide. 2010 Candidate, Vacancy and Client Contact Matching. Candidate, Vacancy and Client Contact Matching Candidate, Vacancy and Client Contact Matching Page 1 Last Revised: January 2010 Candidate, Vacancy and Client Contact Matching Features and Related Topics: 1 Candidate and Vacancy Matching:...3 2 Candidate

More information

UNIVERSITY OF NEBRASKA AT OMAHA COLLEGE OF ARTS AND SCIENCES GUIDELINES ON REAPPOINTMENT, PROMOTION, AND TENURE

UNIVERSITY OF NEBRASKA AT OMAHA COLLEGE OF ARTS AND SCIENCES GUIDELINES ON REAPPOINTMENT, PROMOTION, AND TENURE UNIVERSITY OF NEBRASKA AT OMAHA COLLEGE OF ARTS AND SCIENCES GUIDELINES ON REAPPOINTMENT, PROMOTION, AND TENURE Preamble The faculty of the College of Arts and Sciences hereby establishes these standards

More information

Using bibliometric maps of science in a science policy context

Using bibliometric maps of science in a science policy context Using bibliometric maps of science in a science policy context Ed Noyons ABSTRACT Within the context of science policy new softwares has been created for mapping, and for this reason it is necessary to

More information

Compass Interdisciplinary Virtual Conference 19-30 Oct 2009

Compass Interdisciplinary Virtual Conference 19-30 Oct 2009 Compass Interdisciplinary Virtual Conference 19-30 Oct 2009 10 Things New Scholars should do to get published Duane Wegener Professor of Social Psychology, Purdue University Hello, I hope you re having

More information

Research and Digital Game- based Play: A Review of Martha Madison

Research and Digital Game- based Play: A Review of Martha Madison Research and Digital Game- based Play: A Review of Martha Madison White Paper Compiled by Anne Snyder, Ph.D. Second Avenue Learning is a certified women- owned business, specializing in the creation of

More information

Bibliometrics and Transaction Log Analysis. Bibliometrics Citation Analysis Transaction Log Analysis

Bibliometrics and Transaction Log Analysis. Bibliometrics Citation Analysis Transaction Log Analysis and Transaction Log Analysis Bibliometrics Citation Analysis Transaction Log Analysis Definitions: Quantitative study of literatures as reflected in bibliographies Use of quantitative analysis and statistics

More information

Ensemble Methods. Knowledge Discovery and Data Mining 2 (VU) (707.004) Roman Kern. KTI, TU Graz 2015-03-05

Ensemble Methods. Knowledge Discovery and Data Mining 2 (VU) (707.004) Roman Kern. KTI, TU Graz 2015-03-05 Ensemble Methods Knowledge Discovery and Data Mining 2 (VU) (707004) Roman Kern KTI, TU Graz 2015-03-05 Roman Kern (KTI, TU Graz) Ensemble Methods 2015-03-05 1 / 38 Outline 1 Introduction 2 Classification

More information