Pushing UH-CRIS through research evaluation tricks and results Aija Kaitera Project manager (Research information system) Acknowledgements: Seppo Saari Project Manager (International Evaluations) Maria Forsman Chief Information Specialist (Social Sciences, Education and Law) www.helsinki.fi/yliopisto
University of Helsinki in brief Established in Turku 1640, moved to Helsinki 1828 Main tasks: research, teaching and societal interaction Bilingual (Finnish and Swedish), tuition also provided in English 11 faculties 37 000 degree students, 32 000 continuing education and Open University students. 8 670 employees, including 4 845 researchers and teachers (3 800 Full time). Total turnover 2010: EUR 630 million Operates on four campuses in Helsinki and 17 other locations throughout Finland 2
Research volume Outcome: 11 700 publications - 6 600 scientific articles and book - 5 100 with more societal impact Research 405,3M 65 % Societal interaction 89,2M 14 % Education 132,2M 21 % 2011 Rankings: Times Higher Education 91 Shanghai Ranking Consultancy 74 3
Focus areas of research The basic structure, materials and natural resources of the world Basic structure of life Changing environment clean water Thinking and learning human being Well-being and safety Clinical research Exact thinking Language and culture Social justice Globalisation and changing society 4
CRIS-project at the University of Helsinki Initiative after 2005 research evaluation Data per departments -> poor results for research network type activity Pilot at the Medical Faculty 2007 University level project and procurement 2008 Kick off June 2010 First deadline January 31, 2011 Annual reporting for the University Research evaluation
UH CRIS TUHAT Research information system UH corporate management UH Intra & Internet Researcher s homepage CV Legacy data UH DATAWAREHOUS Pure4 UH digital archive HELDA User DB Personnel DB Student DB Project DB Publication metadata sources Web of Science, ArXiv National DBs Researcher s own inforamtion
TO THE TOP AND OUT TO SOCIETY INTERNATIONAL EVALUATION OF RESEARCH AND DOCTORAL TRAINING AT THE UNIVERSITY OF HELSINKI, 2010-2012 Seppo Saari www.helsinki.fi/yliopisto 7.11.2 011 7
PARTICIPATION CATEGORIES 1 Researcher community is internationally on the cutting edge of its field. 2 The research is of high quality but the researcher community does not have strong international status or well-defined breakthrough yet. 3 The research differs from main stream research and is considered exceptional due to specific, justified features of the research field/area and tradition which need to be taken into account in the evaluation. 4 The research represents an innovative opening in the field. 5 The research has a highly significant societal impact. Seppo Saari
Evaluation aspects Scientific quality Scientific significance Societal impact Processes and good practices in leadership and management International, national and cross-sectoral collaboration and mobility, including contacts with society Innovativeness Future significance Seppo Saari
Participating Researcher Communities 136 Researcher Communities 5 857 Researchers Category 1 2 3 4 5 Tot % Persons Bio 12 3 5 5 2 27 20 1360 Hum 10 6 6 4 2 28 20 1090 Med 12 4 0 4 2 22 17 1160 Nat scs 12 3 0 6 2 23 17 1120 Soc scs 11 8 5 6 6 36 26 1230 Total 57 24 16 25 14 136 100 Persons 3050 1090 570 710 540 5960 Seppo Saari
Evaluation material Seppo Saari A. Questionnaire 1. Focus and quality of the RC s research (publications, activities, statistics and bibliometric report) 2. Practises and quality of doctoral training (including information about doctoral dissertations supervised or under supervision 3. The societal impact of research and doctoral training (including statistics and lists of other scientific activity ) 4. International and national research collaboration and researcher mobility 5. Operational conditions 6. Leadership and management in the researcher community 7. External competitive funding of the RC 8. The RC s strategic action plan for 2011 2013 B. Evaluation of the category of the RC in the context of entity of the evaluation material (1 8) C. Description of how the RC members contributed the compilation of the materials. D. How are the focus areas in research presented in the RC s evaluation documents?
Evaluation process (1) Evaluation method developed iteratively Spring 2010 Summer 2010 Dec 2010 Jan 2011 Feb 2011 March 2011 Announcement Decision to use UH CRIS for data collecting. Data collecting criteria announced Deadline for data entering First draft of compilation-pdf announced More data management by research communities and evaluation office
Evaluation process (2) April 2011 May 2011 June 2011 Aug 2011 Aug 2011 Fall 2011 March 2012 Publication data is ready for Bibliometrical analyses Publication data is enriched with Web of Science data by the University of Leiden (CWTS) Panelists receive first bibliometrical analyses based on WoS data Library produces bibliometrical analyses for Sociel Sciences and Humanities Panel visits Rector s funding allocation Panel report is published
RC in portal http://tuhat.halvi.helsinki.fi/portal/
RC material Researchers CRIS ID
RC material xml report template
RC material compiling pdf s (1)
RC material data for bibliometrical analyses CWTS mapping checked and approved by UH using CWTS interface
RC material compiling pdf s (2)
Problems in compiling the data System was implemented in June 2010. UH CRIS creates pdf with system providers logo and other fixed features that are irrelevant to panelists. Count of publication massive, use as little space as possible. The Identity of Researcher Community members. Time period - people who were long gone were included in the evaluation. Analyses and listings were created RCspecifically. WOS identification required for CWTS. Additionally: Activities were collected from all, real importance for selected fields of science. Data inconsistent, particularly legacy data. Solutions All data had to be imported/entered to new system. Create reports in word, modify and print as pdf. New rendering format ordered from Atira. RC s were manually mapped to CRIS-data. Legacy people were created in the system manually. Create report templates separately for all 136 RC s. WoS UT added to publications. Activities analyses was manually changes to only include total volume per research community. Manual changes required.
Alternative bibliometric analyses at the Helsinki University Library Maria Forsman 7.9.2011 "Research Evaluation and Bibliometrics" www.helsinki.fi/yliopisto
Bibliometrical analyses At the University of Helsinki, researchers opinions have been taken into account during the international evaluation of research and doctoral training. Web of Science analyses made by University of Leiden (CWTS). HULib alternative analyses for those research groups and fields of sciences that are weakly represented in the Web of Science. Maria Forsman
Background Critics towards bibliometric especially citation analyses - among researchers of social sciences, humanities and computer science. The researchers feel that these analyses are often unfair: not a good enough picture of publishing in these fields of sciences. The weak coverage of social sciences and humanities in citation databases Web of Science and Scopus In humanities and social sciences monographs are still an important publishing forum, and monographs don t include in these databases, and citations to them only occasionally. In computer science conference papers are the main publishing forum. Maria Forsman
What did the UH library do? Number of authors co-authorship, co-operation Language international & national publishing Journals quality ranking by research communities Book publishers quality ranking by research communities Conferences quality ranking by research communities Maria Forsman
Maria Forsman Period 2005-2010 HULib analyses: an example of all research groups under evaluation from TUHAT data
Maria Forsman Language of publications HULib analyses: all research groups & TUHAT data
The top journals of HU researchers (more than 100 articles 2005-2010) Journals 2005 2006 2007 2008 2009 2010 Grand T otal Helsingin Sanomat 63 85 95 95 86 125 549 Duodecim 41 59 52 53 65 52 322 Physical Review Letters 36 56 56 43 57 47 295 Physical Review D : Particles, Fields, Grav 42 34 41 41 74 46 278 EFSA Journal 26 15 27 52 126 246 Teologinen Aikakauskirja 33 46 41 34 42 33 229 Suomen lääkärilehti 37 45 41 28 40 33 224 Hufvudstadsbladet 52 34 39 39 25 30 219 Arctos : Acta Philologica Fennica 37 72 29 27 30 1 196 Virittäjä 22 40 33 19 39 28 181 Tieteessä tapahtuu 26 35 23 28 25 29 166 Yliopisto : Helsingin yliopiston tiedelehti 29 44 29 14 21 18 155 Parnasso 25 27 28 26 20 22 148 Atmospheric Chemistry and Physics 17 14 26 23 30 31 141 Lakimies 26 17 28 27 17 22 137 PLoS One 2 5 42 44 44 137 Idäntutkimus 19 17 25 25 23 23 132 Arkhimedes 25 32 23 12 13 23 128 Diabetologia 20 29 14 23 15 20 121 Proceedings of the National Academy of S 13 13 15 34 26 19 120 Boreal Environment Research 27 2 32 4 39 8 112 Nature Genetics 7 4 6 19 25 50 111 Maria Forsman Hiidenkivi 23 20 16 20 15 16 110
Top journals, Norway ranking and in comparison - the Finnish publication forum ranking (coming soon) Journals Grand T otal Norway ranking Julkaisuforum Helsingin Sanomat 549 Duodecim 322 1 Physical Review Letters 295 2 2 Physical Review D : Particles, Fields, Gravitation and Cosmology 278 2 EFSA Journal 246 Teologinen Aikakauskirja 229 1 Suomen lääkärilehti 224 1 Hufvudstadsbladet 219 Arctos : Acta Philologica Fennica 196 1 1 Virittäjä 181 2 2 Tieteessä tapahtuu 166 Yliopisto : Helsingin yliopiston tiedelehti 155 Parnasso 148 Atmospheric Chemistry and Physics 141 1 2 Lakimies 137 1 1 PLoS One 137 1 2 Idäntutkimus 132 1 Arkhimedes 128 Diabetologia 121 1 2 Proceedings of the National Academy of Sciences of the United States of America 120 2 2 Boreal Environment Research 112 1 2 Nature Genetics 111 2 2 Hiidenkivi 110 Maria Forsman
Notes on future needs TUHAT database is most important for bibliometric analyses and reserch evaluation the data must be correct and reliable - Affiliations - Subject description Possibilities in the future: network analyses (coauthorship, co-affiliationship, co-word) The Book Citation Index in Web of Science (coming 2011) Book review analyses Publish or Perish now individual researchers what about analysing research groups? More qualitative analyses! Maria Forsman
Thank you! UH CRIS: http://www.helsinki.fi/tuhat/ UH IR: http://helda.helsinki.fi UH advertisement: http://www.youtube.com/universityofhelsinki