Evaluation of Research Quality (VQR )

Transcription

1 Evaluation of Research Quality (VQR ) Criteria for the evaluation of research products of the Group of evaluation experts Area 14 (GEV 14) 1. Intro The purpose of this document is to present in the clearest way all the criteria GEV of Area 14 along with the guidelines provided by ANVUR for VQR will take for granted in evaluating any received product, in order to allow researchers ad structures to make aware choices about products to be sent, and in the same time guarantee maximum transparency in the evaluation process. This document is divided in 4 parts: the first one delimits the area through reference to scientificdisciplinary areas (from now and on called SSD) ( 2). The second one specifies all rules of GEV organization and sub-groups GEV is divided into ( 3). The third one establishes some basic method points and sets all criteria respected by GEV in order to evaluate a research ( 4). In the last part, information about conflict of interest will be provided ( 5). 2. Area Delimitation Are included in the Area 14 Social and Political Sciences all the followings 1 : SPS/01 - FILOSOFIA POLITICA SPS/02 - STORIA DELLE DOTTRINE POLITICHE SPS/03 - STORIA DELLE ISTITUZIONI POLITICHE SPS/04 - SCIENZA POLITICA SPS/05 - STORIA E ISTITUZIONI DELLE AMERICHE SPS/06 - STORIA DELLE RELAZIONI INTERNAZIONALI SPS/07 - SOCIOLOGIA GENERALE SPS/08 - SOCIOLOGIA DEI PROCESSI CULTURALI E COMUNICATIVI SPS/09 - SOCIOLOGIA DEI PROCESSI ECONOMICI E DEL LAVORO SPS/10 - SOCIOLOGIA DELL'AMBIENTE E DEL TERRITORIO SPS/11 - SOCIOLOGIA DEI FENOMENI POLITICI SPS/12 - SOCIOLOGIA GIURIDICA, DELLA DEVIANZA E MUTAMENTO SOCIALE 1 Source: Decreto Ministeriale 29 luglio 2011 n

2 SPS/13 - STORIA E ISTITUZIONI DELL'AFRICA SPS/14 - STORIA E ISTITUZIONI DELL'ASIA 3. GEV Organization 3.1 Sub-GEV GEV will be organized into two subgroups: I) Sub-area of Political Studies, comprehending those following SSD SPS/01 - FILOSOFIA POLITICA SPS/02 - STORIA DELLE DOTTRINE POLITICHE SPS/03 - STORIA DELLE ISTITUZIONI POLITICHE SPS/04 - SCIENZA POLITICA SPS/05 - STORIA E ISTITUZIONI DELLE AMERICHE SPS/06 - STORIA DELLE RELAZIONI INTERNAZIONALI SPS/13 - STORIA E ISTITUZIONI DELL'AFRICA SPS/14 - STORIA E ISTITUZIONI DELL'ASIA; II) Sub-area of Sociological Studies, comprehending those following SSD SPS/07 - SOCIOLOGIA GENERALE SPS/08 - SOCIOLOGIA DEI PROCESSI CULTURALI E COMUNICATIVI SPS/09 - SOCIOLOGIA DEI PROCESSI ECONOMICI E DEL LAVORO SPS/10 - SOCIOLOGIA DELL'AMBIENTE E DEL TERRITORIO SPS/11 - SOCIOLOGIA DEI FENOMENI POLITICI SPS/12 - SOCIOLOGIA GIURIDICA, DELLA DEVIANZA E MUTAMENTO SOCIALE. Members of Sub-GEV Political Sciences are: Laura Bazzicalupo Università degli Studi di Salerno (Coordinator) Maurizio Cotta Università degli Studi di Siena Alessandro Ferrara Università degli Studi di Roma Tor Vergata Simona Forti Università del Piemonte Orientale Jean-pierre Gaudin Sciences-Po Paris Gianfranco Pasquino Università degli Studi di Bologna Members of Sub-GEV Social Sciences are: Milly Buonanno Università degli Studi di Roma Sapienza Sandro Cattacin Université de Genève Roberto Cipriani Università degli Studi di Roma Tre (Coordinator) 2

3 Ivo Colozzi Università degli Studi di Bologna (President) Antonietta Mazzette Università degli Studi di Sassari Emilio Reyneri Università degli Studi di Milano Bicocca Giovanna Rossi Università degli Studi di Milano Sacro Cuore 3.2. Assignment of products in the GEV The assignment of products to sub-gev will happen with reference with the SSD specified by the evaluated subject and transmitted by the structures. The very meaning of the SSD associated to the product that can also be different from the belonging SSD of the evaluated subject is to be given to the GEV with more competence of the field in accordance with the evaluated subject. If ever the products by an author are assigned to more than one GEV, identical evaluation criteria will be adopted agreed among all concerned GEVs. In order to do this, interested GEV Presidents will establish specific Inter-Area Consensus Groups Functioning rules of GEV Convocation of GEV occurs 15 days ahead of any reunion. The reunion is convoked by the President and he is the one in charge also for deciding the subject-matters. Any decision in the GEV is taken upon simple majority of presents. To participate voting it is not mandatory to be physically present, although computerized presence is indispensable. To GEV reunions with secretariat functions and with no allowance of vote will be present GEV Assistant provided by ANVUR, Brigida Blasi. At the end of every reunion, a report will come out in Italian, and an abstract with approved decisions in English language. Report and Abstract will circulate through GEV members and shall be approved via or using the SW provided by CINECA. 4. Evaluation Mix All the following products, types a), b), c), e) listed in the section 2.3 of the Bando ANVUR VQR , shall be evaluated using the method called informed peer review: a) articles on journals; b) books, chapters of books, conference proceedings having an ISBN code only; c) critical editions, translations, scientific comments and encyclopedia contents; e) multimedia products, shows and organized expos, data bases and software, thematic maps only if joined by publications meant to consent an adequate evaluation of the product. For detailed information on single products, refer to FAQ on the ANVUR website. All articles published on journals index-linked in the databases Web of Science of Thomson Reuters (WoS) and/or Scopus will be experimentally evaluated also upon bibliometric base, in order to study the correlation between those two methods Peer Review Products will be sent to two external referees or evaluated, if competence conditions do exist and in absence of conflicts of interest, by the GEV using the same procedures and the same reviser s sheet 3

4 meant for external experts. The selection of external referees, Italian and foreigners, assumed its relevant finalities of public utility, is based on the principle of loyal institutional cooperation, and is held by correctness criteria, objectivity and impartiality. Every component of GEV is obliged to guarantee anonymity of experts whom any product evaluation is assigned to. Information acquired and aimed for the selection of referees are to be used for VQR only. The selection of external referees will be made avoiding conflicts of interest between the referees themselves and author(s) and/or the affiliation structure. Furthermore, independence of referees will be guaranteed, and focus will be kept on affiliation structure, scientific collaboration and where possible nationality. To minimize conflicts of interests, referees operating out of national boundaries will be preferred. Where possible, 2 different members of GEV will select two peer revisers separately Selection of peer revisers The GEV has the aim to involve preferably external referees having an international research profile, and a high-profile curriculum witnessed by a large amount of publications within the last years in the top international publication channels of the field, a significant received citation number and a suitable competence in the specific evaluation area. GEV will prepare a list of all external referees setting minimal standards of scientific quality impact on the international scientific community and evaluation experience. Great attention will concern anonymity of revisers, both in writing the list of referees and in the operative phase of evaluation. For what concerns the first phase, the President of GEV will consult the list of revisers of his/her own area available on CINECA, and will ask GEV components through SubGEV coordinators, to suggest a significant number of experts able to satisfy criteria on the previous point. The President of GEV will gather all indications and information given on the base of a shared sheet and along with sub-gev Coordinators will modify the CINECA list with integrations and cancellations. This modification process of the list will go on for the whole evaluation time, on the basis of needs emerged after the transmission of products by structures Grant of Class of Merit VQR on the base of peer evaluation Referees evaluation is based on a sheet called scheda revisore (reviser s sheet) made by GEV, consisting of multiple choice questions and a blank for a limited number of words. The GEV will transform all indications on that sheet in one of the 4 final classes of merit. Publications discovered to be pieces of plagiarism or even partial fraud will be evaluated negative (-2). In case of not matching evaluations coming from peer revisers, subgev will create within itself a Gruppo di Consenso (Consensus Group) with the aim of purposing to the GEV the final score of the product in object, using the consensus report method. The Gruppo di Consenso can also refer to the advice of another expert. Anyway, responsibility in concluding evaluations is GEV s. 4

5 Products published on journals To determine the final class of merit for articles published on journals, the GEV will follow the method called informed peer review, which consists in using several different elements of evaluation, possibly independent one from another and nevertheless putting them together. The responsibility of the final evaluation is of the whole GEV, and is based on peer evaluations and for articles published on journals index-linked in Web of Science of Thomson Reuters (WoS) and/or Scopus on bibliometric data coming from an algorithm counting (on different standards upon issue date) both received citation number and bibliometric indicator of the host journals. For articles appeared on Italian journals, the GEV has set a rating of all of them that is to be published on the ANVUR website within March 20 th,2012, and articulated into 3 classes of merit, using the following procedure: 1. Request to societies and scientific national committees of Area 14 to propose a classification of magazines in at least 3 classes, along with criteria used to create it. 2. Critical exam of all proposals coming from more than one anonymous qualified external expert operating out of the national boundaries. 3. Shipping back to societies and scientific committees any observation and proposal about different collocation of journals, along with all motivations in order to make counter deductions possible. 4. Approval of the final classification by the GEV Monographs and book chapters All monographs and chapters of books will be sent to 2 external peer revisers, selected through the same criteria mentioned in paragraph Precautions in sending He/She who sends monographs and/or book chapters is required to give on the product sheet provided by CINECA all information related to prizes, reviews on national and international press, translations and any other detail considered useful in the evaluation process Grant of levels of merit VQR on the base of peer evaluation Also for monographs, the revisers evaluation will be based on a sheet called scheda revisore (reviser s sheet) made by GEV, consisting of multiple choice questions and a blank for a limited number of words. GEV will transform all indications on that sheet in one of the 4 final classes of merit. In case of not-matching evaluations coming from peer revisers, the subgev will create an internal Gruppo di Consenso having the task to propose to the GEV the final score of the product object of odd judgment by external experts, through the method called consensus report. The Gruppo di Consenso can also refer to the expertise of a third proficient expert. Anyway the responsibility of the final evaluation is GEV s, that shall consider: all peer evaluations; characteristics of the publishing series the monograph was issued by; the existence of a publishing committee; transparency in publishing decision; national and international spread of the publisher s products; opinions and reviews on international press; and any other indication useful in evaluating quality and effects of the work. 5

6 5. Resolution of conflicts of interests GEV members will not evaluate nor give out to other GEV members or external experts: a. Products they are authors of, or co-authors b. Products whose authors or co-authors are relatives and similar, until forth cousins. c. Products coming from Universities those members have or have had job relationships or official collaborations with, including the affiliation to research agencies, since 1/1/2007. d. Products coming from research agencies under surveillance of MIUR or other public or private subjects volunteering VQR those members have or have had job relationships or official collaborations with, including the affiliation to research agencies, since 1/1/2007. In cases recalling point 1) conflict of interests does exist: i. If the structure has a permanent internal structuration (territorial or disciplinary i.e. local section of research agency, institute, department) limitedly to products presented by this same structure ii. If the structure does not have a permanent internal structuration (territorial or disciplinary i.e. local section of research agency, institute) in relation to any product iii. If the structure has several internal hierarchical levels, (i.e. more than one institute included in the same department) conflict of interests exist to the lowest level (i.e. GEV members affiliated to different institutes of the department are in conflict of interests in relation with products presented by authors coming from the same institute only). In case of conflict of interests, GEV President will give charge of all evaluation procedures to another GEV member, not having conflict of interests. In case of conflict of interests concerning the President, the products assignment will be operated by the VQR Coordinator or other person he/she decides to entrust. 6