How to measure excellence: the case of the Swiss NCCRs Stefan Bachmann, SNSF
Contents 1. The SNSF in summary 2. National Centres of Competence in Research (NCCR) 3. Evaluation, progress reporting and output data collection 4. Conclusions, open questions
1. The Swiss National Science Foundation in Summary
Key features of the SNSF _ Supports scientific research _ Founded in 1952, under private law _ Largest Swiss institution for research support _ Research body of the Swiss Federation _ Regular Federal funding: CHF 656 million (2010) = EUR 505 million
Funding Opportunities _ Project Funding _ Individual Funding _ Conference Grants _ Publication Grants _ Research Programmes (NRP, NCCR) _ International Co-operation
National Research Council Structure Division I: Humanities and Social Sciences Division II: Mathematics, Natural and Engineering Sciences Division III: Biology and Medicine Division IV: Targeted Research Section National Research Programmes (NRP) Section National Centres of Competence in Research (NCCR) Commitees: Interdisciplinary Research, Individual Funding (ad personam), International Co-operation
Use of funds (I) 2009 38 % 25 % Humanities and Social Sciences Mathematics, Natural and Engineering Sciences 37 % Biology and Medicine Total: CHF 707 million
Use of funds (II) 2009 Project Funding 2 % 10 % 2 % Individual Funding 23 % 63 % Scientific Conferences, Publication Grants, International Co-operation National Research Programmes (NRP) National Centres of Competence in Research (NCCR) Total: CHF 707 million
2. The National Centres of Competence in Research (NCCR)
Main features _ National network of excellence with a strong local anchorage at the home institution _ Substantial funding by SNSF ( CHF 2 5 million/year) _ Support by the home institution (financial contributions, basic infrastructure, services for the management) _ Contract partners: SNSF, home institution, NCCR director _ High scientific, financial and organisational autonomy close controlling by SNSF _ Long-term perspective (max. duration: 3 * 4 = 12 years)
Aims of the NCCRs _ Maintaining and strengthening of Switzerland s international competitiveness in strategically important research areas _ Funding of innovative and cutting edge research _ Initiating research at the interface between disciplines _ Bridging the gap between basic science and application _ Initiating structural effects and their institutionalising
Intended structural effects _ Definition of research priorities within Swiss research institutions, in particular in the home institutions of the NCCRs (e.g. new or re-orientation of professorships, infrastructure, equipment) _ Promotion of collaboration with other research institutions in Switzerland and abroad _ Strengthening the pool of young researchers and promotion of career opportunities for women _ Stimulating partnerships between universities and public/private sector
Key figures _ 1 st group from 2001: 14 NCCRs _ 2 nd group from 2005: 6 NCCRs _ 3 rd group from 2010: 8 NCCRs Period 2005 2008 (in CHF) SNSF Funding 257 Mio. (35%) Self-funding Home Institutions 115 Mio. (16%) Self-funding groups 266 Mio. (36%) Third-party Funding 93 Mio. (13%)
The NCCRs in Switzerland Molecular Oncology MaNEP Plant Survival MICS LIVES Chemical Genetics Biology Affective Sciences Nanoscale Science Mediality Iconic Democracy MUST Criticism Structural Quantum Science Biology and Technology Climate Co-Me Neuro FINRISK Kidney Transcure Robotics IM2 North-South Trade Regulation Quantum Photonics Synaptic Bases of Mental Diseases
Current NCCRs (I) 1 st group (from 2001) _ Climate (Bern) _ CO-ME: Computer Aided and Image Guided Medical Interventions (ETH Zurich) _ FINRISK: Financial Valuation and Risk Management (Zurich) _ Genetics (Geneva) _ IM2: Interactive Multimodal Information Management (Idiap, Martigny) _ MaNEP: Material with Novel Electronic Properties (Geneva)
Current NCCRs (II) _ MICS: Mobile Information and Communication Systems (EPF Lausanne) _ Molecular Oncology (EPF Lausanne) _ Nanoscale Science (Basle) _ Neuro: Neural Plasticity and Repair (Zurich) _ North South (Bern) _ Plant Survival (Neuchâtel) _ Quantum Photonics (EPF Lausanne) _ Structural Biology (Zurich)
Current NCCRs (III) 2 nd group (from 2005) _ Affective Sciences: Emotions in individual behaviour and social processes (Geneva) _ Democracy: Challenges to Democracy in the 21 st Century (Zurich) _ Iconic Criticism: The Analysis of Image Processes (Basle) _ Mediality: Historical Perspectives (Zurich) _ SESAM: Swiss Etiological Study of Adjustment and Mental Health (Basle) _ Trade Regulation: From Fragmentation to Coherence (Bern)
Current NCCRs (IV) 3 rd group (from 2010) _ TransCure: From transport physiology to identification of therapeutic targets (Bern) _ Chemical biology (Geneva/EPF Lausanne) _ LIVES: Overcoming vulnerability life course perspectives (Lausanne/Geneva) _ Kidney.CH: Kidney control of homeostasis (Zurich) _ Robotics (EPF Lausanne) _ SYNAPSY: Synaptic bases of mental diseases (EPF Lausanne/Geneva/ Lausanne) _ QSIT: Quantum science and technology (ETH Zurich) _ MUST: Molecular Ultrafast Sciences and Technology (ETH Zurich/Bern)
3. Evaluation, Progress Reporting and output data collection
Reporting High autonomy of the NCCR (start new projects, stop projects, reallocation of funding, internal rules) One review panel for each NCCR (6 8 international experts and 2 3 members of SNSF Research Council) One progress report per year One site visit per year (2 days) Report of review panel to SNSF Research Council (with recommendations) Decision of Research Council on release of next year s funding (with or without conditions)
The Annual Progress Report Progress Report Text part: on research, KTT, education, advancement of women communication, structural impact, management, reaction to last year s recommendations of the panel No page limitations ( as short as possible, as long as necessary ) normally 70-100 pages Lists and statistics part: Personnel Education/training (running /completed PhDs, postdocs) Scientific output Other forms of output Cooperations with third parties Financial situation and finance plan
Output indicators Scientific output: Papers with peer review Papers without peer review Scientific articles in anthologies Books Reports Presentations at congresses/fairs Other forms of output: Services, patents, licences, start-up/spin-off companies, prototypes/demonstrators, other forms of knowledge and technology transfer
The Annual Site Visit The 2-day programme is fixed by SNSF in close cooperation with the NCCR and in consideration of specific wishes by the panel. Discussions of the panel with the NCCR management Presentations (of overall development, of projects, of other aspects) Poster sessions, lab or field visits, group discussions Discussion with representatives of the Home Institution (Rector or Vice-Rector, Dean) Internal deliberation of the panel Short feedback of the panel speaker to the NCCR management and the representatives of the Home Institution Panel report (6-8 pages) with findings and recommendations. Written by SNSF staff, verified by panel (no paper work for the panel members)
Evaluation of proposals Proposals are evaluated by several international Selection Panels for the main scientific fields (Call 2008: Natural Sciences & Engineering / Social Sciences and Humanities / Life Sciences / Medicine) Rating (recommended/not recommended) among the proposals of the respective field based on written proposal, on the presentation by and discussion with the 3-5 main applicants of the proposed NCCR and on cross-comparison of the proposal in the respective field Selection of best proposals by the Research Council, no cross-comparison between the fields Final selection and approval by State Secretariat for Science and Education based on science policy criteria
4. Conclusions, Open Questions
Conclusions State of the art at SNSF: Measuring excellence is not a mathematical problem! no quantitative ranking lists, no bibliometric measures Assessments are made by international peers (panel), based on qualitative and quantitative information as well as on their on-site discussions and impressions international community as benchmark New: NCCRs have to define their short-/long-term goals for KTT, Education, Advancement of Women, Communication in Year 1 (strategy papers) own goals and ambitions as benchmark For decision on continuation (between 4-year-phases): Evaluation of the past development compare current state of the NCCR with starting situation
Open questions Inclusion of bibliometric methods, impact factors, h-index etc. is in discussion but: avoid taking the same indicators and metrics for all disciplines, focus on the metrics of the field may help cross-comparisons within the field, but not between the fields Alternative approach: Ask the researchers How do you define quality, what are your criteria? Monitoring by international peers will avoid too unambitious goal setting Self-defined goals seem to have a much higher acceptability by the researchers than top-down imposed indicators and benchmarks
Thank you for your attention www.snf.ch nccr@snf.ch sbachmann@snf.ch