Pushing UH-CRIS through research evaluation tricks and results

Transcription

1 Pushing UH-CRIS through research evaluation tricks and results Aija Kaitera Project manager (Research information system) Acknowledgements: Seppo Saari Project Manager (International Evaluations) Maria Forsman Chief Information Specialist (Social Sciences, Education and Law)

2 University of Helsinki in brief Established in Turku 1640, moved to Helsinki 1828 Main tasks: research, teaching and societal interaction Bilingual (Finnish and Swedish), tuition also provided in English 11 faculties degree students, continuing education and Open University students employees, including researchers and teachers (3 800 Full time). Total turnover 2010: EUR 630 million Operates on four campuses in Helsinki and 17 other locations throughout Finland 2

3 Research volume Outcome: publications scientific articles and book with more societal impact Research 405,3M 65 % Societal interaction 89,2M 14 % Education 132,2M 21 % 2011 Rankings: Times Higher Education 91 Shanghai Ranking Consultancy 74 3

4 Focus areas of research The basic structure, materials and natural resources of the world Basic structure of life Changing environment clean water Thinking and learning human being Well-being and safety Clinical research Exact thinking Language and culture Social justice Globalisation and changing society 4

5 CRIS-project at the University of Helsinki Initiative after 2005 research evaluation Data per departments -> poor results for research network type activity Pilot at the Medical Faculty 2007 University level project and procurement 2008 Kick off June 2010 First deadline January 31, 2011 Annual reporting for the University Research evaluation

6 UH CRIS TUHAT Research information system UH corporate management UH Intra & Internet Researcher s homepage CV Legacy data UH DATAWAREHOUS Pure4 UH digital archive HELDA User DB Personnel DB Student DB Project DB Publication metadata sources Web of Science, ArXiv National DBs Researcher s own inforamtion

7 TO THE TOP AND OUT TO SOCIETY INTERNATIONAL EVALUATION OF RESEARCH AND DOCTORAL TRAINING AT THE UNIVERSITY OF HELSINKI, Seppo Saari

8 PARTICIPATION CATEGORIES 1 Researcher community is internationally on the cutting edge of its field. 2 The research is of high quality but the researcher community does not have strong international status or well-defined breakthrough yet. 3 The research differs from main stream research and is considered exceptional due to specific, justified features of the research field/area and tradition which need to be taken into account in the evaluation. 4 The research represents an innovative opening in the field. 5 The research has a highly significant societal impact. Seppo Saari

9 Evaluation aspects Scientific quality Scientific significance Societal impact Processes and good practices in leadership and management International, national and cross-sectoral collaboration and mobility, including contacts with society Innovativeness Future significance Seppo Saari

10 Participating Researcher Communities 136 Researcher Communities Researchers Category Tot % Persons Bio Hum Med Nat scs Soc scs Total Persons Seppo Saari

11 Evaluation material Seppo Saari A. Questionnaire 1. Focus and quality of the RC s research (publications, activities, statistics and bibliometric report) 2. Practises and quality of doctoral training (including information about doctoral dissertations supervised or under supervision 3. The societal impact of research and doctoral training (including statistics and lists of other scientific activity ) 4. International and national research collaboration and researcher mobility 5. Operational conditions 6. Leadership and management in the researcher community 7. External competitive funding of the RC 8. The RC s strategic action plan for B. Evaluation of the category of the RC in the context of entity of the evaluation material (1 8) C. Description of how the RC members contributed the compilation of the materials. D. How are the focus areas in research presented in the RC s evaluation documents?

12 Evaluation process (1) Evaluation method developed iteratively Spring 2010 Summer 2010 Dec 2010 Jan 2011 Feb 2011 March 2011 Announcement Decision to use UH CRIS for data collecting. Data collecting criteria announced Deadline for data entering First draft of compilation-pdf announced More data management by research communities and evaluation office

13 Evaluation process (2) April 2011 May 2011 June 2011 Aug 2011 Aug 2011 Fall 2011 March 2012 Publication data is ready for Bibliometrical analyses Publication data is enriched with Web of Science data by the University of Leiden (CWTS) Panelists receive first bibliometrical analyses based on WoS data Library produces bibliometrical analyses for Sociel Sciences and Humanities Panel visits Rector s funding allocation Panel report is published

14 RC in portal

15

16 RC material Researchers CRIS ID

17 RC material xml report template

18 RC material compiling pdf s (1)

19 RC material data for bibliometrical analyses CWTS mapping checked and approved by UH using CWTS interface

20 RC material compiling pdf s (2)

21 Problems in compiling the data System was implemented in June UH CRIS creates pdf with system providers logo and other fixed features that are irrelevant to panelists. Count of publication massive, use as little space as possible. The Identity of Researcher Community members. Time period - people who were long gone were included in the evaluation. Analyses and listings were created RCspecifically. WOS identification required for CWTS. Additionally: Activities were collected from all, real importance for selected fields of science. Data inconsistent, particularly legacy data. Solutions All data had to be imported/entered to new system. Create reports in word, modify and print as pdf. New rendering format ordered from Atira. RC s were manually mapped to CRIS-data. Legacy people were created in the system manually. Create report templates separately for all 136 RC s. WoS UT added to publications. Activities analyses was manually changes to only include total volume per research community. Manual changes required.

22 Alternative bibliometric analyses at the Helsinki University Library Maria Forsman "Research Evaluation and Bibliometrics"

23 Bibliometrical analyses At the University of Helsinki, researchers opinions have been taken into account during the international evaluation of research and doctoral training. Web of Science analyses made by University of Leiden (CWTS). HULib alternative analyses for those research groups and fields of sciences that are weakly represented in the Web of Science. Maria Forsman

24 Background Critics towards bibliometric especially citation analyses - among researchers of social sciences, humanities and computer science. The researchers feel that these analyses are often unfair: not a good enough picture of publishing in these fields of sciences. The weak coverage of social sciences and humanities in citation databases Web of Science and Scopus In humanities and social sciences monographs are still an important publishing forum, and monographs don t include in these databases, and citations to them only occasionally. In computer science conference papers are the main publishing forum. Maria Forsman

25 What did the UH library do? Number of authors co-authorship, co-operation Language international & national publishing Journals quality ranking by research communities Book publishers quality ranking by research communities Conferences quality ranking by research communities Maria Forsman

26 Maria Forsman Period HULib analyses: an example of all research groups under evaluation from TUHAT data

27 Maria Forsman Language of publications HULib analyses: all research groups & TUHAT data

28 The top journals of HU researchers (more than 100 articles ) Journals Grand T otal Helsingin Sanomat Duodecim Physical Review Letters Physical Review D : Particles, Fields, Grav EFSA Journal Teologinen Aikakauskirja Suomen lääkärilehti Hufvudstadsbladet Arctos : Acta Philologica Fennica Virittäjä Tieteessä tapahtuu Yliopisto : Helsingin yliopiston tiedelehti Parnasso Atmospheric Chemistry and Physics Lakimies PLoS One Idäntutkimus Arkhimedes Diabetologia Proceedings of the National Academy of S Boreal Environment Research Nature Genetics Maria Forsman Hiidenkivi

29 Top journals, Norway ranking and in comparison - the Finnish publication forum ranking (coming soon) Journals Grand T otal Norway ranking Julkaisuforum Helsingin Sanomat 549 Duodecim Physical Review Letters Physical Review D : Particles, Fields, Gravitation and Cosmology EFSA Journal 246 Teologinen Aikakauskirja Suomen lääkärilehti Hufvudstadsbladet 219 Arctos : Acta Philologica Fennica Virittäjä Tieteessä tapahtuu 166 Yliopisto : Helsingin yliopiston tiedelehti 155 Parnasso 148 Atmospheric Chemistry and Physics Lakimies PLoS One Idäntutkimus Arkhimedes 128 Diabetologia Proceedings of the National Academy of Sciences of the United States of America Boreal Environment Research Nature Genetics Hiidenkivi 110 Maria Forsman

30 Notes on future needs TUHAT database is most important for bibliometric analyses and reserch evaluation the data must be correct and reliable - Affiliations - Subject description Possibilities in the future: network analyses (coauthorship, co-affiliationship, co-word) The Book Citation Index in Web of Science (coming 2011) Book review analyses Publish or Perish now individual researchers what about analysing research groups? More qualitative analyses! Maria Forsman

31 Thank you! UH CRIS: UH IR: UH advertisement: