ASSESSING THE GENERIC COMPETENCES ACQUIRED BY STUDENTS GRADUATING FROM ITALIAN UNIVERSITIES ANVUR S FINAL REPORT

Transcription

1 ASSESSING THE GENERIC COMPETENCES ACQUIRED BY STUDENTS GRADUATING FROM ITALIAN UNIVERSITIES ANVUR S FINAL REPORT Rome, 11 March 214 Reviewed and finalized on 3 July 214 1

2 Table of Contents 1. Introduction with a concluding comment Pilot test numbers TECO features Results Conclusions Reasons and criteria for the TECO pilot test Reasons for the pilot test Formal reasons Substantial reasons Criteria for the experiment Cost of the pilot test and its coverage Processes, timeline and phases of the pilot test on the generic competences of Italian graduating students Appointment of the Committee of Guarantors and the Working Group Creation and completion of the National Project Office (NPO), national control room of the experiment Selection of the Universities participating to the pilot test Local governance: the Professors - Institutional Coordinators (ICP), the Administrative Institutional Coordinators (ICA) and the Lead Scorers (LSC) Selection and adaptation of the test The CAE-ANVUR contract The Performance Task (PT) in TECO The 2 multiple choice questions or Selected Response Questions (SRQ) in TECO Adaptation of the CLA+ and its transformation into TECO Presentation of the experiment to the stakeholders via seminars Set-up of the technological platforms and collection of the contextual variables Translation and conciliation of the texts Focus group and cognitive laboratory at the University of Camerino Validation of the translation after the focus group Test Administration Test administration management by CINECA and CAE, up to the release of individual results Training of Lead Scorers (LSCs) and Scorers (SCs), and scoring of the open-response test (PT) Data checking and cleaning Public presentation of the outcomes of the experiment Main facts emerging from the experiment The regularity index, R, in University studies The TECO participation index, P TECO passes the feasibility test in Italy Comparability of TECO results and scores between Italian graduating students and similar student populations in the rest of the world The specifically Italian problem of the two cultures

3 4.6 The top performers Simple and multiple correlations between TECO results and contextual variables The influence of the family s socio-cultural condition Other social and family information Supports for studying and individual merit Students self-assessment of the competences they have acquired Initial estimates and corrections for contextual diversities Initial estimates and corrections for the self-selection bias Externalities of merit Overview in 2 points of the main outcomes of the TECO pilot test Overview of the outcomes of the TECO pilot test as regards geographic areas TECO Test on the generic competences of graduating students PT module (open-ended response) Excerpt from the SRQ (closed-response items) module Index of Tables Other Tables Index of Annexes available upon request Bibliography

4 This ANVUR Report has been elaborated, based on the data received up to 31 January 214, by Fiorella Kostoris Padoa Schioppa with the assistance of Valentina Testuzza (ANVUR), Marzia Foroni (MIUR), Massimo Carfagna (CRUI) and Paola Costantini (ANVUR), the extraordinary cooperation of Emanuela Reale, and the high competence and generous help of Alessio Ancaiani, Alberto Ciolfi and Irene Mazzotta (ANVUR). ANVUR is grateful to all who have contributed to make this Report so informative. ANVUR also wishes to thank the financial sponsors, guarantors, working group members, experts, translators, scorers, all those in charge of local governance, and all participants to the seminars on the TECO test. Pre-Feasibility Working Group Guido Franco Amoretti (Univ. of Genoa); Gabriele Anzellotti (Univ. of Trento); Annamaria Poggi (Univ. of Turin); Emanuela Reale (CERIS, CNR); Roberto Ricci (INVALSI); Paolo Sestito (Bank of Italy); Vincenzo Zara (Univ. of Salento-Lecce). Financial sponsors INVITALIA; Fondazione S. Paolo; Fondazione Cariplo; Fondazione Caripuglia for the University of Salento; Region of Friuli Venezia Giulia for the University of Udine. Guarantors Alfonso Caramazza (Universities of Harvard and Trento); Jan Levy (AHELO, OCSE); Piero Cipollone (World Bank, Washington DC); Roberto Ricci (INVALSI). Experts Claudio Borri (Univ. of Florence); Alfonso Caramazza (Universities of Harvard and Trento); Alberto Mantovani (Univ. of Milan); Giorgio Parisi (Univ. of Rome La Sapienza); Annamaria Poggi (Univ. of Turin); Emanuela Reale (CERIS, CNR); Roberto Ricci (INVALSI); Emanuela Stefani (CRUI); Vincenzo Zara (Univ. of Salento, Lecce); Doris Zahner (CAE, New York); Barbara Frabboni, Alessandro Lodi, Maurizio Moreo, Mauro Motta and Francesca Pruneti (CINECA). Institutional Coordinators in each of the 12 Universities participating in the pilot test Eliana Baici (Univ. of Eastern Piedmont); Carlotta Berti Ceroni (Univ. of Bologna); Carlo Busacca (Univ. of Messina); Tiziana Catarci (Univ. of Rome La Sapienza); Giuseppe De Luca (Univ. of Milan); Ettore Felisatti (Univ. of Padua); Stefano Manetti (Univ. of Florence); Riccardo Martina (Univ. of Naples); Aurelio Simone (Univ. of Rome Tor Vergata); Maurizio Trifone (Univ. of Cagliari); Fabio Vendruscolo (Univ. of Udine); Vincenzo Zara (Univ. of Salento, Lecce, replaced by Alessandra Chirco on 6/5/213). 4

5 Administrative Institutional Coordinators in each of the 12 Universities participating in the pilot test Clorinda Capria (Univ. of Messina); Vincenzo De Marco (Univ. of Florence); Elena De Sanctis (Univ. of Bologna), Emanuela Della Valle (Univ. of Milan); Eusebio Giandomenico and Domenico Genovese (Univ. of Rome Tor Vergata); Gabriella Gianfrate (Univ. of Salento, Lecce); Giuseppa Locci (Univ. of Cagliari); Alessandra Missana (Univ. of Udine); Rosalba Natale (Univ. of Rome La Sapienza); Cristina Stocco (Univ. of Padua); Maurizio Tafuto (Univ. of Naples); Andrea Turolla (Univ. of Eastern Piedmont). Members of the Coordination Committees in the 12 Universities participating in the pilot test Carmen Aina (Univ. of Eastern Piedmont); Enrica Amaturo (Univ. of Naples); Giuseppe Pio Anastasi (Univ. of Messina); Marisa Arcisto (Univ. of Eastern Piedmont); Federica Atzeni (Univ. of Cagliari); Eliana Baici (Univ. of Eastern Piedmont); Riccardo Banfo (Univ. of Eastern Piedmont); Simonetta Bartolucci (Univ. of Naples); Achille Basile (Univ. of Naples); Marco Antonio Bazzocchi (Univ. of Bologna); Bruno Bertaccini (Univ. of Florence); Carlotta Berti Ceroni (Univ. of Bologna); Dimitri Boatta (Univ. of Rome La Sapienza); Renato Brandimarti (Univ. of Bologna); Pierfrancesco Brunello (Univ. of Padua); Carlo Busacca (Univ. of Messina); Clorinda Capria (Univ. of Messina); Settimio Carmignani Caridi (Univ. of Rome Tor Vergata); Tiziana Catarci (Univ. of Rome La Sapienza); Marcantonio Catelani (Univ. of Florence); Alessandra Chirco (Univ. of Salento, Lecce); Giuseppe Cirino (Univ. of Naples); Sonia Consalvo (Univ. of Rome Tor Vergata); Marcello Corvo (Univ. of Rome Tor Vergata); Edoardo Matias Diaz Crescitelli (Univ. of Rome Tor Vergata); Giuseppe De Luca (Univ. of Milan); Vincenzo De Marco (Univ. of Florence); Lucia De Nitto (Univ. of Salento, Lecce); Elena De Sanctis (Univ. of Bologna); Arturo De Vivo (Univ. of Naples); Stefano Del Giudice (Univ. of Udine); Maria Vittoria Dell'Anna (Univ. of Salento, Lecce); Emanuela Dellavalle (Univ. of Milan); Paolo Di Francesco (Univ. of Rome Tor Vergata); Francesca Dragotto (Univ. of Rome Tor Vergata); Carla Faralli (Univ. of Bologna); Silvia Fedeli (Univ. of Rome La Sapienza); Ettore Felisatti (Univ. of Padua); Patrizio Gabrielli (Univ. of Rome Tor Vergata); Domenico Genovese (Univ. of Rome Tor Vergata); Marianna Gensabella (Univ. of Messina); Eusebio Giandomenico (Univ. of Rome Tor Vergata); Gabriella Gianfrate (Univ. of Salento, Lecce); Stefano Giordani (Univ. of Rome Tor Vergata); Fiorella Giusberti (Univ. of Bologna); Elisa Latino (Univ. of Salento, Lecce); Giuseppa Locci (Univ. of Cagliari); Mara Lucisano (Univ. of Milan); Stefano Manetti (Univ. of Florence); Aldo Manzin (Univ. of Cagliari); Marella Maroder (Univ. of Rome La Sapienza); Federico Masini (Univ. of Rome La Sapienza); Giorgio Massacci (Univ. of Cagliari); Carla Massidda (Univ. of Cagliari); Marco Mazzotta (Univ. of Rome Tor Vergata); Moreno Meneghetti (Univ. of Padua); Danilo Merlo (Univ. of Messina); Angela Maria Mezzasalma (Univ. of Messina); Giuseppe Micheli (Univ. of Padua); Alessandra Missana (Univ. of Udine); Rosalba Natale (Univ. of Rome La Sapienza); Giuseppe Novelli (Univ. of Rome Tor Vergata); Anna Nozzoli (Univ. of Florence); Pellegrino Palumbo (Univ. of Naples); Monica Paolini (Univ. of Milan); Mauro Patrone (Univ. of Eastern Piedmont); Cecilia Pennetta (Univ. of Salento, Lecce); Franco Peracchi (Univ. of Rome Tor Vergata); 5

6 Elio Pietro Perrone (Univ. of Rome Tor Vergata); Domenico Petrazzuoli (Univ. of Naples); Barbara Pietrobono (Univ. of Rome Tor Vergata); Luciano Pinotti (Univ. of Milan); Flavio Pressacco (Univ. of Udine); Marina Quartu (Univ. of Cagliari); Maria Carla Re (Univ. of Bologna); Ludovico Rella (Univ. of Florence); Egidio Robusto (Univ. of Padua); Eugenia Rossi Di Schio (Univ. of Bologna); Filomena Russo (Univ. of Rome Tor Vergata); Piero Salatino (Univ. of Naples); Francesco Scerbo (Univ. of Rome Tor Vergata); Maria Eugenia Schininà (Univ. of Rome La Sapienza); Sabina Simeone (Univ. of Rome Tor Vergata); Aurelio Simone (Univ. of Rome Tor Vergata); Annamaria Spada (Univ. of Milan); Cristina Stocco (Univ. of Padua); Maurizio Tafuto (Univ. of Naples); Piero Toma (Univ. of Salento, Lecce); Maurizio Trifone (Univ. of Cagliari); Andrea Turolla (Univ. of Eastern Piedmont); Maria Antonietta Vanoni (Univ. of Milan); Gabriella Vanotti (Univ. of Eastern Piedmont); Fabio Vendruscolo (Univ. of Udine); Iacopo Versari (Univ. of Bologna); Cesare Voci (Univ. of Padua); Vincenzo Zara (Univ. of Salento, Lecce). Lead Scorers in each of the 12 Universities participating in the pilot test, monitored by the Lead of Lead Scorers Roberto Ricci and his INVALSI working group coordinated by Cristina Stringher, and Doris Zahner (CAE) for closed-response scoring and final superscoring Eliana Baici (Univ. of Eastern Piedmont); Giuseppe De Luca (Univ. of Milan); Silvia Fedeli (Univ. of Rome La Sapienza); Roberto Giuntini (Univ. of Cagliari); Fiorella Giusberti (Univ. of Bologna); Franz Heinrich Kohnke (Univ. of Messina); Stefano Manetti (Univ. of Florence), Riccardo Martina (Univ. of Naples); Vittorio Rocco (Univ. of Rome Tor Vergata, replaced by Gianluca Cubadda on 29/1/214); Egidio Robusto (Univ. of Padua); Carlo Sempi (Univ. of Salento, Lecce); Fabio Vendruscolo (Univ. of Udine). Scorers Antonio Acconcia (Univ. of Naples); Giovanna Adinolfi (Univ. of Milan); Ivan Rossano Adorno (Univ. of Salento, Lecce); Enzo Vinicio Alliegro (Univ. of Naples); Alessandra Allini (Univ. of Naples); Leonardo Altieri (Univ. of Bologna); Eliana Baici (Univ. of Eastern Piedmont); Cristian Balducci (Univ. of Bologna); Luciano Maria Barone (Univ. of Rome La Sapienza); Cecilia Bartuli (Univ. of Rome La Sapienza); Sergio Beraldo (Univ. of Naples); Nicola Boccella (Univ. of Rome La Sapienza); Vanna Boffo (Univ. of Florence); Maria Broccardo (Univ. of Rome La Sapienza); Maria Fiorenza Caboni (Univ. of Bologna); Gilberto Calderoni (Univ. of Rome La Sapienza); Paolo Calvosa (Univ. of Naples); Luigi Campanella (Univ. of Rome La Sapienza); Daniele Cananzi (Univ. of Rome La Sapienza); Enrica Caporali (Univ. of Florence); Settimio Carmignani Caridi (Univ. of Rome Tor Vergata); Marcantonio Catelani (Univ. of Florence); Paola Catenaccio (Univ. of Milan); Marta Cavagnaro (Univ. of Rome La Sapienza); Cristiana Cianitto (Univ. of Milan); Chiara Cini (Univ. of Rome La Sapienza); Paolo Clavenzani (Univ. of Bologna); Fabrizio Consorti (Univ. of Rome La Sapienza); Antonio Contestabile (Univ. of Bologna); Stefano Cordiner (Univ. of Rome Tor Vergata); Alessandro Corsini (Univ. of Rome La Sapienza); Marco 6

7 Costa (Univ. of Bologna); Ilaria Cutica (Univ. of Milan); Antonio D'Alessandro (Univ. of Rome La Sapienza); Marco De Amici (Univ. of Milan); Giuseppe De Luca (Univ. of Milan); Giovanna Del Gobbo (Univ. of Florence); Maria Vittoria Dell'Anna (Univ. del Salento, Lecce); Paolo Di Francesco (Univ. of Rome Tor Vergata); Giancarlo Fabrizi (Univ. of Rome La Sapienza); Giuseppe Familiari (Univ. of Rome La Sapienza); Silvia Fedeli (Univ. of Rome La Sapienza); Fabio Ferlazzo (Univ. of Rome La Sapienza); Lea Ferrari (Univ. of Padua); Marzia Freo (Univ. of Bologna); Fabiana Fusco (Univ. of Udine); Elisa Maria Galliani (Univ. of Padua); Dora Gambardella (Univ. of Naples); Roberta Gemmiti (Univ. of Rome La Sapienza); Marco Gherghi (Univ. of Naples); Francesca Giofrè (Univ. of Rome La Sapienza); Marco Giunti (Univ. of Cagliari); Roberto Giuntini (Univ. of Cagliari); Fiorella Giusberti (Univ. of Bologna); Laura Grassini (Univ. of Florence); Valentina Grion (Univ. of Padua); Franz Heinrich Kohnke (Univ. of Messina); Sandro Landucci (Univ. of Florence); Agostina Longo (Univ. of Rome La Sapienza); Roberta Maeran (Univ. of Padua); Marco Maffei (Univ. of Naples); Franco Maggi (Univ. of Milan); Elisa Magnani (Univ. of Bologna); Stefano Manetti (Univ. of Florence); Gianluigi Mangia (Univ. of Naples); Marina Marino (Univ. of Naples); Riccardo Martina (Univ. of Naples); Marcella Martinelli (Univ. of Bologna); Barbara Mazza (Univ. of Rome La Sapienza); Dora Melucci (Univ. of Bologna); Manuela Merli (Univ. of Rome La Sapienza); Nadia Netti (Univ. of Naples); Laura Nota (Univ. of Padua); Carlo Maria Orlandelli (Univ. of Bologna); Francesco Paoli (Univ. of Cagliari); Guido Parravicini (Univ. of Milan); Esterina Pascale (Univ. of Rome La Sapienza); Fulvia Patella (Univ. of Rome Tor Vergata); Mauro Patrone (Univ. of Eastern Piedmont); Elisabetta Petrucci (Univ. of Rome La Sapienza); Maria Cristina Piccirilli (Univ. of Florence); Luciano Piergiovanni (Univ. of Milan); Bruna Pieri (Univ. of Bologna); Luciano Pinotti (Univ. of Milan); Flavio Pressacco (Univ. of Udine); Marina Pugnaletto (Univ. of Rome La Sapienza); Carla Rampichini (Univ. of Florence); Alberto Reatti (Univ. of Florence); Paolo Ricciardi (Univ. of Rome La Sapienza); Paola Ricciulli (Univ. of Rome La Sapienza); Francesca Ripari (Univ. of Rome La Sapienza); Egidio Robusto (Univ. of Padua); Vittorio Rocco (Univ. of Rome Tor Vergata); Stefano Romegnoli (Univ. of Padua); Maria Novella Romenelli (Univ. of Florence); Silvia Salini (Univ. of Milan); Vincenzo Scalzo (Univ. of Naples); Carlo Sempi (Univ. del Salento, Lecce); Roberto Serpieri (Univ. of Naples); Teresa Maria Sgaramella (Univ. of Padua); Luca Stefanutti (Univ. of Padua); Alberto Tamburini (Univ. of Milan); Luca Tardella (Univ. of Rome La Sapienza); Arjuna Tuzzi (Univ. of Padua); Gabriella Vanotti (Univ. of Eastern Piedmont); Barbara Vari (Univ. of Milan); Fabio Vendruscolo (Univ. of Udine); Luigi Ventura (Univ. of Rome La Sapienza); Silvia Vida (Univ. of Bologna); Antonio Villari (Univ. of Messina). Translators Maria Alessandra Scalise (INVALSI) and Andrea Ferrari (CAPSTAN, Brussels). 7

8 Speakers at the information seminars (12 seminars between 29 November 212 and 18 February 213) Seminar in NOVARA (Univ. of Eastern Piedmont) on 29/11/212 Cesare Emanuel (Univ. of Eastern Piedmont); Carmen Aina (Univ. of Eastern Piedmont); Marisa Arcisto (Univ. of Eastern Piedmont); Eliana Baici (Univ. of eastern Piedmont); Loris Barberis (Univ. of Eastern Piedmont); Graziella Berta (Univ. of Eastern Piedmont); Giorgia Casalone (Univ. of Eastern Piedmont); Alberto Cassone (Univ. of Eastern Piedmont); Umberto Dianzani (Univ. of Eastern Piedmont); Giovanni Fraquelli (Univ. of Eastern Piedmont); Anna Invernizzi (Associazione Industriali Novara); Lucrezia Songini (Univ. of Eastern Piedmont); Annamaria Torazzo (Univ. of Eastern Piedmont); Andrea Turolla (Univ. of Eastern Piedmont); Mario Valletta (Univ. of Eastern Piedmont). Seminar in LECCE (Univ. of Salento) on 1/12/212 Vincenzo Zara (Univ. of Salento, Lecce); Domenico Laforgia (Univ. of Salento, Lecce); Gabriella Gianfrate (Univ. of Salento, Lecce); Carlo Margiotta (Univ. of Salento, Lecce). Seminar in FLORENCE (Univ. of Florence) on 11/1/213 Alberto Tesi (Univ. of Florence); Marco Bellandi (Univ. of Florence); Bruno Bertaccini (Univ. of Florence); Marcantonio Catelani (Univ. of Florence); Mario Curia (Confindustria Florence); Vincenzo De Marco (Univ. of Florence); Stefano Manetti (Univ. of Florence); Anna Nozzoli (Univ. of Florence); Giacomo Poggi (Univ. of Florence); Emanuela Stefani (CRUI); Vincenzo Zara (Univ. of Salento, Lecce). Seminar in UDINE (Univ. of Udine) on 18/1/213 Cristiana Compagno (Univ. of Udine); Marina Brollo (Univ. of Udine); Paolo Ceccon (Univ. of Udine); Derna Del Stabile (Interna); Francesco Marangon (Univ. of Udine); Alessandra Missana (Univ. of Udine); Roberto Molinaro (Regione Friuli Venezia Giulia); Roberto Siagri (Eurotech); Andrea Tabarron (Univ. of Udine); Alberto Toffolutti (Confindustria Udine); Fabio Vendruscolo (Univ. of Udine). Seminar in MESSINA (Univ. of Messina) on 21/1/213 Francesco Tomasello (Univ. of Messina); Daniela Baglieri (Univ. of Messina); Ivo Blandina (Confindustria Messina); Selena Gasperini (Univ. of Messina); Alessandro Italiano (Univ. of Messina); Maria Enza La Torre (Univ. of Messina); Anna Murdaca (Univ. of Messina); Agatina Scarcella (Univ. of Messina). Seminar in ROME (Univ. of Rome La Sapienza) on 23/1/213 Luigi Frati (Univ. of Rome La Sapienza); Tiziana Catarci (Univ. of Rome La Sapienza); Silvia Fedeli (Univ. of Rome La Sapienza); Maurizio Flammini (Federlazio); Pietro Lucisano (Univ. of Rome La Sapienza); Federico Masini (Univ. of Rome La Sapienza); Rosalba Natale (Univ. of Rome La Sapienza). 8

9 Seminar in ROME (Univ. of Rome Tor Vergata) on 24/1/213 Tiziano Lauro (Univ. of Rome Tor Vergata); Giuseppe Novelli (Univ. of Rome Tor Vergata); Settimio Carmignani Caridi (Univ. of Rome Tor Vergata); Francesco De Antoni (Univ. of Rome Tor Vergata); Paolo Di Francesco (Univ. of Rome Tor Vergata); Domenico Genovese (Univ. of Rome Tor Vergata); Eusebio Giandomenico (Univ. of Rome Tor Vergata); Stefano Giordani (Univ. of Rome Tor Vergata); Nathan Levialdi Ghiron (Univ. of Rome Tor Vergata); Gustavo Piga (Univ. of Rome Tor Vergata); Aurelio Simone (Univ. of Rome Tor Vergata); Nicola Vittorio (Univ. of Rome Tor Vergata). Seminar in MILAN (Univ. of Milan) on 28/1/213 Gianluca Vago (Univ. of Milan); Daniela Candia (Univ. of Milan); Giuseppe De Luca (Univ. of Milan); Emanuela Dellavalle (Univ. of Milan); Stefano Forte (Univ. of Milan); Laura Mengoni (Assolombarda); Alberto Meomartini (Assolombarda); Anna Soru (Chamber of Commerce of Milan); Matteo Turri (Univ. of Milan). Seminar in BOLOGNA (Univ. of Bologna) on 31/1/213 Ivano Dionigi (Univ. of Bologna); Marco Antonio Bazzocchi (Univ. of Bologna); Carlotta Berti Ceroni (Univ. of Bologna); Renato Brandimarti (Univ. of Bologna); Carla Faralli (Univ. of Bologna); Gianluca Fiorentini (Univ. of Bologna); Fiorella Giusberti (Univ. of Bologna); Maria Carla Re (Univ. of Bologna); Eugenia Rossi Di Schio (Univ. of Bologna). Seminar in PADUA (Univ. of Padua) on 1/2/213 Giuseppe Zaccaria (Univ. of Padua); Massimo Castagnaro (ANVUR); Ettore Felisatti (Univ. of Padua); Luciano Galliani (Univ. of Padua); Paolo Gubitta (Univ. of Padua); Giampaolo Pedron (Confindustria Veneto); Edgardo Picardi (Univ. of Padua); Stefano Romegnoli (Univ. of Padua); Cesare Voci (Univ. of Padua). Seminar in CAGLIARI (Univ. of Cagliari) on 11/2/213 Giovanni Melis (Univ. of Cagliari); Tommaso Ercoli (Univ. of Cagliari); Sergio Lai (RSSE); Maurizio Trifone (Univ. of Cagliari); Paolo Gubitta (Univ. of Padua). Seminar in NAPLES (Univ. of Naples) on 18/2/213 Massimo Marrelli (Univ. of Naples); Simonetta Bartolucci (Univ. of Naples); Achille Basile (Univ. of Naples); Giuseppe Cirino (Univ. of Naples); Arturo De Vivo (Univ. of Naples); Paola Izzo (Univ. of Naples); Riccardo Martina (Univ. of Naples); Domenico Petrazzuolo (Univ. of Naples); Piero Salatino (Univ. of Naples). 9

10 1. Introduction with a concluding comment Between 212 and 213, the National Agency for the Evaluation of Universities and Research Institutes (ANVUR) carried out an experimental assessment of the generic learning outcomes shown by students graduating from Italian Universities, by means of the TECO test. This pilot test was designed taking as a reference point the OECD feasibility study called AHELO-Assessing Higher Education Learning Outcomes (website testingstudentanduniversit yperformancegloballyoecdsahelo.htm) (AHELO, 213). ANVUR decided to undertake this pilot test for several reasons. - Formal reasons: Legislative Decree no. 19 of 27 January 212 governing the system of Self- Assessment and Periodic Assessment and Accreditation in higher education (hereinafter referred to as AVA) provides for the introduction of a system of initial and periodic accreditation of study courses and Universities; periodic assessment of the quality, efficiency and outcomes of Universities teaching activities; and enhancement of the mechanisms underpinning the selfassessment of the quality and effectiveness of Universities teaching and research activities. Within this framework, the TECO pilot test has the purpose of supplementing the assessment process, via indicators that provide an external evaluation and an instrument of self evaluation on the quality of learning achieved by students during their studies in terms of the generic competences they possess on graduating from University. - Substantial reasons: the principal stakeholders (employers, Universities, students and their families, taxpayers, and the General Government) are interested in an ever improving quality of education in our Universities. The TECO test aims to measure cross-disciplinary competences: the critical thinking needed to solve a problem or to make a decision, the ability to represent and communicate a given fact, and the ability to learn new knowledge related to areas not necessarily connected with the particularities of the scientific discipline being studied. These generic competences are crucial to ensure individuals flexibility and capability to adapt to personal and professional changes occurring throughout a lifetime. Moreover, these competences are not monitored, assessed or certified by Universities precisely because they are not the subject of specific teaching activities; rather, they are part of that intangible baggage that all teachers should pass on by teaching their subject. Almost thirty Universities offered to participate in the TECO pilot test. The following twelve (a predefined limit) were selected: Eastern Piedmont (PO), Padua (PD), Milan (MI), Udine (UD), Bologna (BO), Florence (FI), Rome La Sapienza (RM1), Rome Tor Vergata (RM2), Naples Federico II (NA), Salento (LE), Cagliari (CA) and Messina (ME), so as to have adequate regional representation (4 from the North, 4 from the Centre and 4 from the South plus Islands), to exclude non-multidisciplinary Universities, and to include Universities with a mix of size characteristics. 1

11 The choice was also guided by the preference for Universities with some previous experience in producing or administering tests used to assess the learning outcomes of University students, as well as for those presumed to have superior IT equipment and administrative robustness. In this Report we will refer to Geographic Areas according to this scheme: Geographic Area NORTH CENTRE SOUTH PO + MI + PD + UD BO + FI + RM1 + RM2 NA + LE + ME + CA University CENTRE + NORTH CENTRE-NORTH CENTRE-SOUTH PO + MI + PD + UD + BO + FI + RM1 + RM2 PO + MI + PD + UD + BO + FI RM1 + RM2 + NA + LE + ME + CA PO + MI + PD + UD + BO + FI + RM1 + RM2 + NA + LE + ME + CA In the design of the TECO pilot test, ANVUR established a series of criteria dictated both by the awareness that it was an experiment (tight deadlines, limited budget, voluntary student participation) and by the need to collect as much data as possible (contextual variables) for a more complete understanding of test results: 1. Using the same test for all University courses, to be evaluated in a uniform way with regard to all students, because generic competences are by their nature independent of the specific field of study; they depend on how you study, not on what is being studied. 2. Using a test consisting of a) an open-response part that enables a check of reading ability, the critical analysis of texts and the ability to make coherent decisions therefrom, as well as writing effectiveness and technique, and b) a closed-response part, regarded as preferable to expose the quality of scientific-quantitative reasoning. 3. Identifying eligible students (corresponding to the notion of graduating students), i.e. those entitled to participate in the test if they are in a defined range of progress and maturity along the study path. 4. Limiting the objective to assessing acquired generic competences (the actual level of learning) and not the added value of University education. This implies excluding freshmen from the test but allows providing significant information to the stakeholders with shorter lead-times. In principle, a longitudinal analysis (on the same people at the beginning and at the end of University studies) would be the best choice to determine the added value created by Universities, but this would require a wait of at least 3-4 years. 11

12 5. Using contextual variables, so as to enable filtering out the part of the individual outcomes of the TECO that depend on both individual characteristics of the student population for example of a personal or family nature and collective characteristics for example the rate of growth in the region of origin or the region where the University is located, which induce a more or less high propensity to rapid and successful completion of studies. This allows a statistical estimate of the added value, through the analysis of the residuals of various multiple regressions. 1.1 Pilot test numbers Regarding the administration of the test, it was known that the people entitled to take the TECO were just under 2% of all students from the third and fourth years, excluding courses for the health professions, enrolled in the 12 participating Universities, i.e. a population of 21,872 in academic year In fact, 14,97 people pre-registered for the test including numerous extraneous persons not eligible for the test and, among those eligible and pre-registered, only about 5,9 students actually came to sit the test (see paragraph 3.11). Tables 2.1 and 2.4 (see paragraph 4.1) show that the mean proportion of eligible candidates out of students from the third and fourth year (regularity index, R) and the mean proportion of those who came to sit the test out of those eligible (participation index, P) range very broadly across the 25 Disciplinary groups and the 12 participating Universities (the set of which is indicated with ). These data pose an initial problem. It is difficult to determine the self-selection bias and, more importantly, to adequately correct for it. If it is of a positive type (as will be shown later), it becomes difficult to make certain assertions on the basis of the data observed in the TECO. For instance, the University of Bologna had greater apparent success in the TECO than Eastern Piedmont. This empirical evidence could mean a higher level of learning outcomes for Bologna, or it could be due to the selfselection of students participating in the test: only 13.91% of the Bologna graduating students came to sit the test, compared with 63.4% for Eastern Piedmont. 1.2 TECO features As regards its structure, the TECO test consists of two main modules. In the first Performance Task (PT) module, a fact or an act or a circumstance of realistic nature are presented in a central document, which identifies a theme, with a set of additional, sometimes incoherent empirical pieces of evidence, often exhibiting varying degrees of robustness. Students are encouraged to take an active role in addressing the issue, suggesting a solution or recommending the most appropriate intervention or deciding between several options presenting desirable and less desirable aspects, on the basis of the information provided. The TECO test does not require any particular knowledge; there are no right or wrong answers, only better or worse argued ones, coherent or incoherent ones, solid or weak answers 12

13 on a logical or empirical level, and answers described with greater or lesser efficacy and appropriateness of language. The PT questions are intended to test three aspects: a) Analysis and Problem Solving (APS), b) Writing Effectiveness (WE), c) Writing Mechanics (WM) Each of the three areas receives a score from to 6: in theory, the minimum score in the PT module is therefore while the maximum is 18; in fact, 45 students who received an overall PT score of less than 3 were eliminated from the evaluation, as it is believed that in such circumstances the level of their engagement with the test is so low that such a case is observationally equivalent" to a case of nonparticipation in the test (see paragraph 3.11). In the second module, called Selected Response Questions (SRQ), 2 questions are proposed with the aim to assess a set of competences of different nature, predominantly scientific-quantitative. For these questions, students must choose the correct answer, discarding three distracter answers, on the basis of the information given or inferred from the documentation provided. SRQ questions are intended to test three aspects: a) Critical Reading (CRE) ability; b) Critique an Argument (CA) ability; c) Scientific and Quantitative Reasoning (SQR) ability. Each question receives a score if the answer is incorrect or missing, a 1 score if the answer is correct: the minimum score in the SRQ module is therefore, while the maximum is Results The first element to be underscored when analysing the results is the fact that the TECO seems to have made the grade as regards feasibility, as shown in Table 3.1 (see section 4.3) and in more detail in the Item Analysis Report by CAE (214): the density distribution approximates a normal distribution with a mean of 1 and a standard deviation of 2. An examination of the frequency distributions of the scores for the two modules shows some left asymmetry in the PT component, some in the opposite direction in the SRQ, and a significant difference between males and females (to the detriment of the latter), in particular in the most scientific and quantitative SQR part of the SRQ module. In addition, it should be pointed out (see paragraph 4.4) that in the twin CLA+ test, given to 4,38 graduating students of US colleges, the results are virtually identical to our own, for both mean and quartiles, illustrating superior writing effectiveness and mechanics in young Italians, as well as greater ability to argue and in critical reading, but lower scientific-quantitative reasoning quality. There is a possibility to validly compare graduating students from the USA, from Italy and from various countries in the world, including some that are very different from one another. This is because the generic competences measured by the TECO pilot test and by the OECD feasibility study (AHELO, 213) are all 13

14 assessed with the same CLA-type open-ended response test (see paragraph 4.4). The only aspect that gives cause for concern in the results of the Italian TECO test, compared with the identical American CLA+ test, is clearly shown in the lower part of Table 3.8 (see paragraph 4.4). The correlation, individual by individual, between the scores obtained in the literary part of the test (PT) and in the scientific-quantitative part (SRQ and particularly SQR) in Italy is half that in the United States. This is a first sign of the so-called two cultures existing in our country. Regardless of the average level of competences acquired at the end of University studies by our students, they normally show logic competences that are much more dissociated between the humanistic and scientific domains versus what is observed elsewhere in the world. Looking at the TECO outcomes by Disciplinary groups (see paragraph 4.5 ), while keeping in mind the observations made above as regards the self-selection bias, the best results in the test are obtained in cases of selection on entry to the University, either with a national admission test (Medicine), or with local admission tests utilized for all candidate entrants (Psychology), or where there is individual selfselection (Mathematics-Physics-Statistics), as evidenced by high grades in the school leaving diploma of those who decide for this study field, known to be stingy when awarding grades. There are six groups for which the TECO scores are significantly below the mean of and, unfortunately, the minimum is reached in the Education group. Disciplines of high importance in Italy such as Philosophy, History, Law, and Literature in the humanities-social sciences fields or Biology and Engineering in the scientific field seem to exceed the national mean and/or median, but not significantly so. The analysis of the two cultures continues by examining Table 4.6 (see paragraph 4.5). Observation of how the two parts of the test went the open-ended, more literary part, and the close-ended, more scientific and quantitative part shows that on average the results correlate well in the Disciplinary groups, with a correlation index of.61. Medicine, Mathematics-Physics-Statistics and Psychology are on average stronger than the others in both aspects, while Education and Sociology are on average weaker in both components. However, while both parts of the test are well harmonised for Psychology students (in the sense that the differences in the two test results are not significant), for those in Medicine and Mathematics-Physics-Statistics there is a clear and strong difference between them, with a prevalence for scientific-quantitative logic. Unsurprisingly, the same is true in the Engineering, Architecture, and Chemistry groups. On the contrary, in the humanities, the Philosophy and History groups who perform better in the TECO, surpassing (but barely) the national mean and median show a balance on average between the two components PT and SRQ, which instead is not seen in the Arts and Law groups for which the performance in the first part is significantly higher than in the second. Unfortunately, this is the case also for the Disciplinary groups with below average success in the TECO, starting with the Education group. Lastly, the analysis of the two cultures is concluded with extreme clarity in Table 4.7 (see paragraph 4.5). In the graph, the dotted interpolation line shows the mean correlation between PT and SRQ scores described above. For each Disciplinary group, a continuous light grey line shows the correlation 14

15 at individual level between the two components of the test. As can be seen from the gradients of all these lines, the individual correlation is very low almost everywhere, as the afore-mentioned comparison between Italy and the United States suggested. Tables 4.11, 4.12 and 4.13 (see paragraph 4.5) propose a further analysis of the two cultures displaying the data broken down by University: it can be seen that, in this case, the mean correlation between PT and SRQ is very strong (.93), while that at individual level is still very weak. We now focus on the contextual variables that most seem to 'influence' results on the TECO. The results in terms of simple correlations are presented in Table 6.1 (see paragraph 4.7). Those obtained with multiple correlations (a work by Franco Peracchi) are set out in Table 7.1. The two types of evidence, when the contextual variables considered match, are basically identical even if sometimes the simple correlation appears stronger (or weaker) given the multicollinearity between various regressors (for example between diploma grades and the professional position of the parents, both of which influence the TECO) and obviously it weakens (or becomes stronger) under the all other things being equal condition adopted in the estimation through multiple regression. There is a systematic downwards relationship between the TECO result and the variables age, female gender (versus male) and residence outside the region of the University's location, as well as an upwards relationship relative to the variables time since diploma obtained, coming from a classical studies high school (compared to other types of high schools), average of diploma and University grades, being single (versus married), Italian citizenship and Italian spoken at home (versus non- Italian citizenship and language). Cases where brothers/sisters are also at the University seem instead to be observationally equivalent to cases where they are not equally educated, while the size of the family seems to have a negative effect. Students with more technological equipment perform better on average, as well as those who go on at least one trip per year outside the region. The influence of parents appears in the sense that an absent mother (not father) lowers the TECO, all other things being equal, and having a father employed in a management position (but not a mother) raises it. The effect of the socio-cultural condition is much stronger in simple correlations (see paragraph 4.7), because in multiple regressions that condition affects the test results also through diploma and University grades, as well as in the choice of secondary school. It can be seen, therefore, that some contextual variables such as, for example, family status lose value once others are controlled. This is specifically because family status helps to predict the type of secondary school diploma, the diploma grade, the type of course of study chosen and the mean University grade in addition to directly predicting the results on the TECO test. Therefore, in simple correlations, a high professional and cultural status of the parents (see paragraph 4.7) strongly correlates with success in the TECO: when the mother is a manager or a white collar employee, has a University degree or high school diploma, regardless of the father's position, results above the mean and median are observed; and this applies equally to the father. The absence of at 15

16 least one parent is obviously a deprivation condition, and the worst one much worse than the father or mother being a manual worker, unemployed or unqualified. The TECO score drops if the student also works and the various types of support for students do not compensate for the disadvantages of different kinds affecting those students who usually have recourse to support. The only type of support that helps raise the TECO result seems to be the student collaboration contract (the only one assigned strictly on merit-based criteria and without consideration for the condition of poverty). It is of particular interest to examine the connection (or lack thereof) between the tested students perception of whether they have acquired adequate competences in the course of their University studies and their performance on the TECO, by Disciplinary groups and by Universities. There is no positive correlation between perception that adequate competences have been acquired and test results, while in the lower two quartiles there is a significant difference in results on the TECO in favour of students who gave a negative response to the question on competences acquired at University (see paragraph 4.11). We thus reach the interesting conclusion that students perceptions that they have acquired competences is indicative of the level of customer satisfaction (high, as it turns out, and particularly so in the Southern Universities) but of nothing else of objective character. Carrying on the analysis of the contextual variables, we note a high correlation between quality of the TECO results and scientific quality of teachers for the corresponding courses of study, as indicated by R12 derived from the VQR (see paragraph 4.14). It is hardly surprising, ex post, that the quality of the results of teaching shows a good match with the quality of the results of research. Finally, it is likely that the self-selection bias is positive, given that the diploma grades (VMD) and University grades (VME) of students who came to sit the test are significantly higher than those of eligible students who did not show up and those of ineligible students (see paragraph 4.13). This holds true for all quartiles of the distribution (see paragraph 4.13). In addition, Tables 8.1 and 8.16 (see paragraph 4.13) indicate that the differences between students who came to sit the TECO test and those who did not, in terms of all the contextual variables that are relevant for simple correlations with results on the TECO, systematically induce a positive self-selection bias. Such variables include age, citizenship, off-site condition, gender, marital status, student worker situation, language spoken at home and other languages known. The question hence arises of how much the results of some Universities with a low P index would go down if the participation rate were to increase and all eligible students were to sit the test not just the self-selected ones that cause the TECO results to rise. As a consequence, in such case, the average University grade or diploma grade would fall to the level corresponding to that of all eligible students lower than the level for just those students who came to sit the test. 16

17 1.4 Conclusions The TECO pilot test carried out by ANVUR at 12 Universities was the first ever attempt to assess the level of generic competences acquired by University students in Italy. The focus was on examining reading ability, critical analysis, ability to solve new logical, interpretative or scientificquantitative problems, and communication capabilities as exhibited by graduating students from all study courses. These competences are at least as important as those that are more closely related to specific courses of study (subject-specific competences). The University has a duty to put generic competences at the centre of its educational activities, because they are necessary for greater adaptability to the job market and to present and future life, and therefore essential to increase employability and personal empowerment. The results observed for the pilot test are overall comparable to those observed elsewhere in the world, as summarized in paragraphs 4.15 and 4.16, but they do point to one specifically Italian weakness: the dissociation in our students between literary and scientific logic, which must be recomposed and overcome in a sort of new Renaissance. Such a goal corresponds to a trend seen in the best practices of North and South America and the Far East, and it is strongly advocated in the Guidelines of the last Council of the 47 European Ministers for Higher Education, meeting in Bucharest in

18 2. Reasons and criteria for the TECO pilot test 2.1 Reasons for the pilot test They are of both a formal and substantial nature Formal reasons Since its creation (DPR 76/21) ANVUR is tasked with assessing not only the processes and the inputs of the educational offer, but also the quality of the results and products of management, teaching and research activities, including technological transfer from Universities and research agencies, also at the level of the individual structures of these institutions (Art. 3 c.1 (a)). More explicitly, Art. 3, c.2 (a) adds Assessment concerns, among others... the efficiency and effectiveness of educational activities on the basis of international quality standards, also with reference to students learning outcomes and students successful insertion in the world of work. In the Guidelines set out by EHEA (European Higher Education Area) in 25, 26 and 28, learning outcomes are broken down, according to the level of detail, into knowledge, skill, competence, or (in French) savoir, savoir faire and savoir être. The Italian translation was not only late to arrive (the transposition by the MIUR only took place in December 21, and the corresponding text "The Italian Degrees Framework" is dated January 211), but furthermore, it does not correspond to the wording or the spirit of 28. It interprets the older, previous formula of the 5 Dublin Descriptors (the first two of a specialist nature, the last three of a generic nature). Indeed, in the official Italian version of European Recommendation 28/C111/1, p. 7 (EU, 28), qualifications is translated with the term titoli in e. degrees (which is closer to certificates ) and the three crucial aspects of competences of a generic nature ( critical thinking, problem solving with decision making, ability to communicate ) are illustrated as follows: The first-cycle level certificates may be awarded to students who have the ability to collect and interpret data which are deemed useful to reach independent conclusions, including reflection on social, scientific or ethical issues; who know how to communicate to specialists and non-specialists; who have developed those learning competences that are necessary to undertake further studies with a high degree of autonomy. The two descriptors of a specialist nature are, on the other hand, more correctly identified: The first-cycle level certificates may be awarded to students who have demonstrated knowledge and understanding at post-secondary level in a field of study which, characterized by the use of advanced textbooks, also includes the knowledge of ground-breaking topics in that field of study; who are able to apply their knowledge and who possess the appropriate competences both to conceive and support arguments and to solve problems in their field of study. Subsequently, Law 24/21 (Art. 5, section 3) and Legislative Decree 19/212 initiate the process leading to the integrated Self-Assessment, Periodic Assessment and Accreditation system (AVA) laid down by ANVUR (213). The initial accreditation requirements include the obligation to describe, 18

19 within the descriptive form drawn up annually for each study course ( Scheda Unica Annuale del Corso di Studio, SUA-CdS), among the course objectives, the expected learning outcomes (both specialized and generic), defined for homogeneous disciplinary areas according to the principles initially adopted by the Bergen Conference of European Ministers Responsible for Higher Education (25). Even more significantly, ANVUR (213) included in chapter F.2 on the Periodic accreditation of University sites and courses of study a section F.2.4 on Additional criteria, indicators and parameters for the periodic accreditation of sites and courses, including those (F.2.4.1) concerning achieved learning outcomes (p. 36 of ANVUR, 213). According to the AVA system, therefore, the results actually achieved by University students in terms of both specialist and generic competences must not only be compared with the expected ones, but also certified, because, in the future, they will be considered for the purposes of periodic accreditation and assessment 1. The intention is thus to introduce a novelty within the Italian higher education system, which, thanks to this initiative, becomes aligned with the best practices underway in most countries of North and South America, the Far East and, to a more limited extent, also in Europe. In Europe progress has so far been slower but the calls to accelerate it by the Conferences of the European Ministers Responsible for Higher Education have been increasingly strong and frequent in the last five years. The most recent illustration is the Official Document approved on April 212 in Bucharest by 47 Member Countries of the European Area, and in particular the strategy document Mobility for better learning. Not by chance, the February 214 draft update of the ENQA Standards and Guidelines (ESG, 214 and 214bis) a document resulting from a consensus between ENQA, ESU, EUA, EURASHE, EI, BUSINESSEUROPE and EQAR finally introduces the concept of student-centred learning, stating that it is necessary that the assessment of students reflects this approach This means careful consideration of the design and delivery of study programmes and the assessment of outcome. The achieved learning outcomes are analysed in relation to the intended outcomes. This type of assessment finally shifts the focus from teaching activities to students actual learning outcomes (briefly stated, from teaching to learning), overcoming the traditional approach based solely on identifying the procedural requirements of a formal nature and the inputs rather than the outputs. Consequently, the AVA system combines a form of quality assurance which is attentive to the minimum conditions needed to efficiently foster an educational offer with an innovative approach centred on teaching effectiveness, as measured by actual results and achieved learning outcomes 2. 1 On the subject of assessing students learning outcomes, Ministerial Decree MIUR 47 of 3 January 213 on selfassessment, initial and periodic accreditation and periodic assessment merely adds a provision concerning distance learning (in Annex C): The assessment of students, through progress checks, is in any case also carried out at locations other than the legal site of the University, provided that it takes place in the presence of the student before a committee established in accordance with the applicable legislation. However, this Ministerial Decree does not (yet) set out precise indicators to assess achieved learning outcomes for the purposes of periodic accreditation and assessment of educational activities, because the pilot test, of which this Report gives an account, ended about 13 months later, in mid-march See also the Council of the European Union (214), which steps up its conclusions in the direction of learning outcomes. 19

20 Substantial reasons There are also substantial reasons, no less important than the formal ones, leading ANVUR to give more importance to assessing achieved learning outcomes. There is increasing pressure for this shift in focus from all those who are more interested in the results than in the procedures and inputs of the University system. There are many such stakeholders: - Employers (including in the financial sector), who require an educated workforce and increasingly argue that while it is easy to find a good engineer or a good philosopher because Italian Universities offer excellent candidates and certify their specialist competences with grades it is difficult to find graduates, from any disciplinary field, who have good cross-disciplinary competences (capacity for critical analysis, decision-making, communication and others). These generic skills and competences are essential for businesses and nobody in Italy assesses or certifies them. - The Universities, which educate our young people and would be keen to improve their diagnostics so as to increase the quality of the courses offered, but which must scrape by as best they can, as all Universities, even private ones, are supported by increasingly scarce public funds. - Students and their families, who want to enhance their human capital as a source of cultural wealth and personal satisfaction as well as an asset for employment and future employability in a job market which is constricted locally but extensive globally, in a perspective (also at the personal level) of unpredictability and extreme volatility. - Italian taxpayers and the General Government, their agent in the principal-agent relationship, who quite rightly want to find out about value for money, efficacy in terms of results of the resources that they contribute to University education, and hence demand that the autonomy of Universities go hand in hand with responsibility (or better: accountability) and assessment. This especially in the midst of a crippling crisis, where the public budget is very tight, the burden on those who pay taxes weighs very heavy, and youth unemployment (also for graduates) is increasingly intolerable. Each of these stakeholders is interested in knowing the level of cross-disciplinary competences shown by our University students at the end of their studies. For some, e.g. employers, that information is sufficient: it is not relevant for them to know when during the journey, the young man or woman acquired such competences (whether at kindergarten, at home, at school or during higher education). But for others families, Universities, taxpayers it would be helpful to also know what value added is gained in the final phase of the journey (in that study course or some other? In that University or in that site?). For all these reasons, it is crucial for ANVUR to assess and certify especially the generic competences acquired by University students: their ability to cope with personal and collective problems in socioeconomic and working contexts not known beforehand, making use of previously acquired knowledge, skills and competences in novel situations. In more detail, these generic competences include knowing how to read and discuss a text never seen before, applying critical thinking to it, including in the 2

21 presence of simple charts and graphs or quantitative symbols; the ability to solve new problems, to make decisions quickly and in risky conditions; the ability to communicate effectively orally and in writing, because work is increasingly carried out with other people: with colleagues by team work or with competitors, but also with clients, suppliers and public authorities. Tests to assess these types of generic competences do not exist in Italy at the level of University studies, but are regularly used on other sections of the population. They demonstrate Italy's wellknown weak points, as is very clear from the evidence produced by the OECD with the PISA tests on fifteen-year-olds (PISA 213) on reading and solving simple quantitative problems, and with the PIAAC tests on the adult population (ISFOL, 213), where Italy ranks last among the almost 3 countries tested on the literacy scale and penultimate in numeracy. Such tests are an essential instrument for evaluating generic competences also in the University setting, as demonstrated by their adoption in a rapidly increasing number of countries around the world. With AHELO (Assessing Higher Education Learning Outcomes), the OECD is attempting to use a single identical generic skill test not only for all fields of knowledge, but also for situations as diverse as are those of Colombia, Egypt, Finland, Korea, Kuwait, Mexico, Norway, Slovakia and the United States the participants in a first feasibility study (AHELO, 213) Criteria for the experiment The Working Group on Pre-feasibility (hereinafter WGP), appointed by the ANVUR Governing Board on 29 May 212 with a mandate to give a quick response to the most urgent strategic issues about the opportunity of providing a test for Italian Universities designed to assess the generic competences of their graduating students, concluded its mission 3 by providing to the Agency a number of suggested guidelines concerning the general criteria to pursue. Illustrated with a wealth of detail in the paper Testing the generic competences achieved by students graduating from Italian Universities: reasons, criteria and design choices, published on the ANVUR website on 1 August 212 (Kostoris Padoa Schioppa, 212), these guidelines underpin Resolution No. 65 of 13 August 212 of the Board of Directors of ANVUR (see Annex 1), the go-ahead for starting the experiment. The general criteria listed by the WGP for the test of generic competences are summarised here again for convenience. They are fully endorsed by ANVUR as emerges from the same Protocol of 18 December 212 (ANVUR, 212) 4 and only in a few points they are subsequently partly rearranged or reformulated in terms shown in detail on the next few pages. The main criteria suggested by the WGP are as follows. 3 The WGP started its work on 5 June 212 and completed it 7 weeks later, in July 212. ANVUR is very grateful to all members of the WGP for the high professional quality and perfect timeliness of its contributions. 4 This concerns the procedures, needed actions and consequences resulting from a pilot test to assess the generic competences of Italian graduating students. It includes, among other things, the offer by CINECA (see Annexes 2 and 3) to administer the questionnaire for Italian graduating students in 12 pilot Universities in 213, at no cost to ANVUR. 21

22 1. Using the same test for all University courses, to be evaluated in a uniform way with regard to all students, because horizontal (generic) competences the first but by no means the only ones to be assessed are throughout the world independent of the specific study paths followed; they depend by their nature on how you study, not on what you study. 2. Using a test designed to measure not general culture, but the ability to read and critically analyse texts that may be either exclusively literary or with some quantitative elements, as well as the ability to make coherent decisions from this analysis and communicate its content in written form. To this end, among the tests available at international level, the one considered preferable by the WGP is the Collegiate Learning Assessment (CLA), ( produced by the Council for Aid to Education (CAE), New York, or a derivative thereof. This test was initially created to provide American colleges which educate undergraduates, equivalent to our students graduating from the three-year first cycle with a useful instrument to continuously improve the quality of learning. It was also used in 9 different countries within the framework of AHELO. Ideally, as stated later by the Committee of Guarantors and Experts of ANVUR, the Italian generic competences test (henceforth TECO) should include both an open-response and a closedresponse part because only the first allows to assess writing effectiveness and writing mechanics, while the second is considered preferable to bring out the quality of scientific-quantitative reasoning. 3. Identifying the University population eligible for TECO in a perimeter defined not by age-related requirements (as in the case of the PISA and PIAAC tests), but of requirements related to progress along the study path (as in AHELO). This leads to defining the notion of graduating students (in a broad sense), corresponding to those University students, excluding the ones enrolled in courses for the health professions, who have acquired all basic and characterising study credits in a three-year first-cycle course or at least 12 basic and characterising credits in a single-cycle master course, as required by their study course. In the experimental phase the test should not be mandatory (although it would be desirable, according to the WGP, that all graduating students in each course take it) owing to the short period between announcement of the test and the test sessions in Universities: the recommended date to determine those who meet the requirements and are therefore eligible for the test (also called graduating or regular students) is 1 April. In the first few months of 213, the third criterion was further refined in three aspects: a) graduating students needs to also include those enrolled in the third year of a three-year first-cycle course, who have passed all the basic and characterising study credits except those offered in the second semester of the third year. Students in this category, although in a perfectly regular situation, may not have already completed all the basic and characterising credits by 1 April; b) for both cost reasons and the interest in covering regular students who are not too old, eligibility is limited to students enrolled in the third and fourth year excluding health 22

23 professions (and not all those who meet the study credits requirements mentioned above), for both three-year first-cycle and single-cycle master courses, regardless of the year when they first enrolled; c) the target in terms of student participation to TECO is lowered, as the authorities in charge of each of the Universities taking part in the pilot test are not able to commit to more than 5% participation out of those eligible, for each course of studies. 4. Limiting the TECO test in the pilot phase to graduating students, excluding freshmen. This is for both budgetary constraints, and so as to be able to provide significant information to the stakeholders within short lead-times. A longitudinal analysis on the same people at the beginning and at the end of University studies would be the best choice to determine the value added created by Universities, but this would require a wait of at least 3-4 years. A cross-section performed simultaneously on students entering and exiting University (to avoid having to wait many years for the results) would immediately double the costs of the pilot test and would anyway be only the second best way to measure value added. Therefore, the objective of the pilot test is limited to assessing the level of generic competences acquired by graduating students, plus getting some approximate estimation of the value added obtained in post-secondary studies by using contextual variables in multiple regressions. 5. Using contextual variables, so as to enable filtering out the part of the individual outcomes of the TECO that depend on both personal characteristics of the student population for example of a personal or family nature and collective characteristics for example the rate of growth in the region of origin or the region where the University is located, which induce a more or less high propensity to rapid and successful completion of studies. The purpose of using such variables is to strive to eliminate those observable factors which, in addition to explaining why some students possess more competences at the end of the study path, would also explain why they would appear better at its start. This allows getting a first, approximate idea of the value added created by a particular course of studies in a particular University, as the unexplained residual in a multiple regression using precisely those explanatory variables as regressors. 2.3 Cost of the pilot test and its coverage From the foregoing, it is already clear that the criterion of cost-effectiveness has inspired and shaped all phases of the experiment. To be precise, only 2, was set aside in the ANVUR budget to be spent on TECO in 213. On the one hand, this has meant a cost-saving orientation for several of ANVUR s strategic choices, such as the decisions to limit the experiment to students leaving University (and not also those entering), and to subject only a subset of students from the third and fourth years to only a generic competences test (and not also disciplinary or subject-specific tests). On the other 23

24 hand, it led the Agency to carry out intense fund-raising work. The drive to collect additional funds was remarkably successful thanks to the extraordinary sensitivity and attention to the problem of our young people s competences demonstrated by several banking foundations and other public institutions, such as the Ministry of Cohesion of the Monti Government (via Invitalia, see Annex 4) and the wonderful Autonomous Region of Friuli-Venezia Giulia, which are all gratefully acknowledged here. As a result, the 2, appropriation in the ANVUR 213 budget not only seemed sufficient at the outset to cover all marginal costs of the TECO experiment over a period of 18 months (closing in March 214), but even overabundant, as the additional net expenditure for ANVUR was projected at less than 5, as shown in Table 1. 24

25 3. Processes, timeline and phases of the pilot test on the generic competences of Italian graduating students Upon completion of the above described activities of the WGP in July 212, ANVUR kicked off the actual testing phase on August 13, 212 by setting up a Working Group (hereinafter WG) tasked with implementing a test of generic competences on Italian graduating students, as well as a Committee of Guarantors (CG) charged with international selection and adaptation for Italy of CAE s CLA test or a derivative thereof (Resolution No. 65 of 13 August 212, Annex 1). ANVUR is extremely grateful to the members of these two groups, listed in the preamble of this Report. To understand the main processes implemented in the pilot test, it is worth noting that it is based on both central governance at the level of ANVUR and local governance at the level of the Universities. Both are highly complex. The central governance has a sort of technical secretariat, partly relying on contributions from external collaborators (institutions 5 and individuals 6 ), but mostly on some excellent researchers within ANVUR 7. The WG, the CG, the translators, the financial sponsors, the IT staff at CAE and CINECA also all cooperate with the central governance thank you all very much. The local governance at the level of the Universities participating in the pilot test is just as elaborate. Each one, set up by its Rector, is co-ordinated by a Professor Institutional Coordinator (ICP) assisted by an Administrative Institutional Coordinator (ICA), operating with a Coordination Group where there are in principle representatives of staff and students in all disciplinary Macro-groups. ANVUR is grateful to all those, listed in the preamble of this Report, for the high quality work they generously offered. The phases and the timing of the pilot phase are shown in the annexed timescale (see Table 2), which is almost identical to the one projected before the start of activities (ANVUR, 212) except for a technical initial delay of 2 months compared to the original roadmap. This was due, on the one hand, to the fact that the Board of Directors of ANVUR agreed on the start of the TECO experiment only in mid-august 212, based on the guidelines proposed by the Working Group on Pre-feasibility in July, and, on the other hand, to the summer break in August, still widespread in Italy and often meaning people can be absent for an entire month. As shown in the timescale diagram, the phases are summarised into 15 points, while the times fall into the 18 months of the experiment (from mid- September 212 to mid-march 214). These phases are referred to in paragraphs of this section 3. 5 As regards institutional partners, a special mention goes to the agreement between ANVUR and CRUI (see Annex 5) stipulated on 19 February 213: the Foundation provides the Agency, for the time of the pilot phase, with the technical support needed to foster the conduction of the project; ANVUR, in turn, offers the Foundation the knowledge acquired as a result of the project, with a view to improving the quality of teaching in Universities. 6 Among the external collaborators, ANVUR is particularly grateful to Paola Felli for her extraordinary patience, devotion and professionalism. 7 Of the internal staff thanks especially to Alessio Ancaiani, Alberto Ciolfi and Irene Mazzotta. 25

26 3.1 Appointment of the Committee of Guarantors and the Working Group The Working Group (WG) was tasked with monitoring the entire TECO experiment on Italian graduating students, while the Committee of Guarantors (CG) had to specifically perform a guarantee function in the international selection of the test, its adaptation and translation, its validation and statistical-psychometric analysis. In particular (Minutes of 19 September 212, see Annex 6): 1. ANVUR asked the CG to collaborate in selecting an international test of generic competences and ensuring its transformation into an equivalent Italian test through adaptation and translation (point 1.a of Resolution No. 65) 2. ANVUR asked the WG to collaborate on operational criteria and concrete means and processes for all activities related to the TECO experiment in Italy, starting with the selection of Universities for the pilot test (point 1.b of Resolution No. 65). Note that while ANVUR for the moment promoted only a test of generic and cross-disciplinary competences, the WG and the CG, in a joint session, agreed on their readiness to also proceed to test subject-specific 26

27 competences, if an interest in doing so should be expressed by the academics and Universities involved in the experiment ANVUR asked the WG to collaborate on the dissemination of information on the test and the implications arising from its use to all stakeholders: students, teachers, families, businesses, the public authorities (point 1.c of the Resolution) Creation and completion of the National Project Office (NPO), national control room of the experiment The National Project Office located in ANVUR, in addition to benefiting from influential members of the CG and the WG for the strategic collective functions described above and for others individually assigned, has also benefited from the contributions of a technical secretariat of junior collaborators, indicated in the preamble of this Report, of which the most important, already mentioned, only joined between September and December 213, once released from other institutional activities at the Agency. The central governance of the experiment interacted for the entire 18 months duration of the project with the local governances at the Universities, because one or the other often had to take and implement decisions together, with a view to improving the assessment of the learning outcomes of students in terms of generic competences, and had to do it in a complementary and timely manner. This happened not without technical difficulties, and sometimes with some tension, always overcome by the determination to obtain, together, useful results for the Universities self-assessment 1 (see Annex 9) as well as for other stakeholders. 3.3 Selection of the Universities participating to the pilot test The selection of the Universities participating in the TECO experiment was initiated by a preliminary invitation sent by the President of ANVUR to the Rectors of all Italian Universities at the end of July 8 In the following months, the CBUI (see Annex 7), representing Biology teachers from all the Universities of the pilot phase that have a Biology department, was very active in this sense and ANVUR confirmed its willingness to introduce a subject-specific test in the TECO programme as long as the test would be produced jointly by the biologists concerned. However, for the moment this test has not yet materialised. 9 At the meeting on 7 November 212 (Annex 8), the responsibilities of the members of the WG were further clarified, in their different tasks: each expert was assigned a leadership role with respect to the different stages of implementation of the project and everyone also supports another expert from the Group, in order to problem share. Of particular importance are the activities carried out by four colleagues, members first of the Working Group on Pre-feasibility (WGP) and then of the WG set up in its wake: Anna Maria Poggi, for the great help in contractual relations; Emanuela Reale, who, with extraordinary expertise and commitment, drafted a first part of this Report; Roberto Ricci for translating the texts and monitoring the scoring performed by INVALSI, as well as for his invaluable contribution as a member of the CG; Vincenzo Zara (until his nomination as Rector) for the highly effective coordination of local coordinators. Without the high professionalism of these colleagues and friends, the success of the TECO venture could hardly have been the same. 1 The form attesting the regularity of the CDL was sent to the Directors on 3 October 213 with a request to return it filled out by 3 November

28 The invitation underscored on the one hand the relevance of the results potentially obtainable from the test for the purposes of improving Universities educational offer. On the other hand, it stressed both the total freedom to choose whether or not to participate in the experiment, and, in the case of assent, the need to take financial responsibility for costs at local level related to test activities, as well as the commitment to fulfil various functions of an IT and administrative nature 12. At the same time, the President of ANVUR stated that the Agency would make a selection from the applications received, pursuing objectives of representation and effectiveness and using transparent methods. The representativeness and effectiveness criteria for selecting the Universities were identified by the WG in the autumn months (in the meetings on 19 September, see Annex 6, and 7 November, see Annex 8). The WG, in particular, suggested considering the following elements to guide the selection of Universities for the pilot test: 1. An adequate composition by regional areas (North-West, North-East, Centre, South and Islands), possibly by selecting 2 Universities in the North-West, 2 in the North-East, 4 in the Centre, 4 in the South, of which preferably 2 in the Islands. 2. The exclusion of non-multidisciplinary Universities. 3. The preference, once constraints 1 and 2 have been met, for Universities with some previous experience in producing or administering tests used to assess learning outcomes 13 of University students or, in the absence of this preferential factor, for those with presumably superior information technology equipment and administrative robustness. 4. The inclusion of Universities with a mix of size characteristics. 5. A maximum limit of 12 Universities to be involved in the pilot phase, possibly offering those not selected the opportunity to nevertheless participate in the experiment in some other way for example by autonomously developing and administering, in cooperation with other candidate Universities, a test to assess subject-specific competences, under the coordination of ANVUR 14, however, bearing in such case the corresponding costs Letter of 27 July 212, protocol 938. In fact, some Universities had already applied during the pre-feasibility phase for participation in a pilot test of assessing actual learning outcomes (University of Bologna, University of Salento, University Federico II of Naples, University of Padua, University of Rome - La Sapienza, and University of Udine). Other Universities had informally expressed an interest in the initiative, requesting more information (among others, the IULM University of Milan, the IUAV of Venice, and the University of Cassino). 12 The University must possess a data warehouse, i.e. a system that allows rapid and efficient querying and management of data on students, as well as support offices that can assist with test implementation operations, without adding to the burden of tasks which are already assigned to department secretariats and students. 13 In the OECD AHELO feasibility study (AHELO, 213), the University of Florence had cooperated in the production of the test used in the Engineering Strand, while the Universities of Eastern Piedmont, Udine, Bologna, Rome - La Sapienza, and Naples had their students take part in the Economics Strand test. 14 Regretfully, some excellent Universities, such as, for example, those of Bari, Camerino, Campobasso, Macerata, Perugia and the University for Foreigners of Siena could not be retained to take part in the ANVUR experiment. 15 In fact what happened was that, for example, in the case of Apulia, ANVUR could not take both the Universities of Bari and Salento (both of which had spontaneously applied) into the experiment group, because they both belong to the same region. When, in the spring of 213, the Caripuglia Foundation decided to allocate an extra 5, so that all four Universities in Apulia (including the Polytechnic of Bari and Foggia) in addition to Lecce which had been selected could test their students with no additional cost for them, the other 3 Universities renounced the opportunity. ANVUR is nevertheless grateful to President Castorani of Caripuglia for the generous offer of resources for the TECO experiment, even if it could not be used. 28

29 Almost thirty Universities submitted an application. The following were selected using criteria 1-5 indicated above: Eastern Piedmont (PO), Padua (PD), Milan (MI), Udine (UD), Bologna (BO), Florence (FI), Rome La Sapienza (RM1), Rome Tor Vergata (RM2), Naples Federico II (NA), Salento (LE), Cagliari (CA) and Messina (ME) 16. The University of Camerino offered subsequently to host the focus group and cognitive lab, a crucial step in the conduction of the project (see section 3.9) and ANVUR gratefully accepted this offer. 3.4 Local governance: the Professors - Institutional Coordinators (ICP), the Administrative Institutional Coordinators (ICA) and the Lead Scorers (LSC) In the same letters of 12 October 212, the President of ANVUR invited each participating University to establish a Coordinating Committee within the University, tasked with ensuring liaison between ANVUR and the governing bodies of the University, with reference to TECO activities. A Professor Institutional Coordinator (ICP), assisted by an Administrative Institutional Coordinator (ICA), chaired the Committee. These persons serving in local governance, mentioned in the preamble of this Report and appointed directly by the Rector, played an essential role in the success of the TECO. The ICPs had their first meeting at ANVUR on 16 November 212; a week later, the first meeting of the ICAs was held at the Agency. Each ICP and ICA had several complex tasks (see also Berti Ceroni, 213), including - referring problems and difficulties at the local level to ANVUR; - reporting locally on matters discussed with the other ICPs and ICAs and the decisions taken together with ANVUR at national level; - providing empirical information about the number and characteristics of students eligible for the test, and, more generally, about the contextual variables relevant to the experiment; - discussing and agreeing on the regulations applicable to conduction of the test, on matters such as the percentage of eligible students 17 in each course that should sit the test, or the period in which to administer the test 18 ; - arranging a very detailed analysis, through a specific format provided by ANVUR (see Annex 9), by those responsible for each class (and if necessary for each individual course) in any University in 16 See the application acceptance letters to the Rectors from the President of ANVUR on 12 October 212, protocols 1338 to 1349, and the application rejection letters sent to the Rectors from the President of ANVUR on the same date, protocols 135 to For example, a majority of the ICPs argued that it was not possible to arrange for all eligible students to sit the test, an option that the WG and the CG had, on the other hand, recommended to ANVUR to avoid any form of self-selection bias. It was then agreed to abide by the rule of a minimum of 5% of the eligible students in each course, a target which was in fact mostly not met, except at the Universities of Udine and Eastern Piedmont. 18 Initially, ANVUR wanted the test to be held between June and July 213, whereas the ICPs and the LSCs asked and obtained to bring it forward, starting from the second half of May (see paragraph 3.11). 29

30 which very irregular paths emerge or particularly weak participation in the test by students of the third and fourth year, in order to launch a thorough self-assessment of the University in this regard. In a letter dated 15 January 213, Protocol 92, the President of ANVUR asked the Rectors of the twelve participating Universities to appoint a Lead Scorer (LSC) to whom to entrust the difficult task of coordinating the scorers of their University and the final scoring of the open-response part of the test, entrusted to 11 scorers 19 (see also paragraph 3.13). Having regard to the strategic role of the LSCs for the TECO, ANVUR required that they preferably be professors with great academic authority, so that they could interact effectively with the various institutional components of the University's teaching activity. The LSC could possibly be the same person as the ICP, but the idea of two different teachers taking on these responsibilities was certainly not discouraged. The extraordinary ability demonstrated by the twelve LSCs in the training of the scorers and in scoring the tests will be further illustrated: ANVUR is very grateful to them for the great professionalism demonstrated. 3.5 Selection and adaptation of the test The CAE-ANVUR contract Based on the Working Group for Pre-feasibility s recommendation to use CAE s Collegiate Learning Assessment (CLA) for the pilot test, ANVUR had already implicitly decided in August 212 in favour of this US provider. Subsequently, the Committee of Guarantors agreed on the advisability, moreover attested to in the international literature, to foresee a test for graduating students with a dual component open-response and closed-response so as to more effectively assess diversified aspects of the ability to read, write, analyse, argue critically, and solve problems featuring both qualitative and quantitative elements. In a group with a very high level of expertise and professionalism, the Norwegian Jan Levi, President of AHELO, put his great experience at the service of ANVUR to explain to his colleagues from the CG (who unanimously agreed) that it was much preferable to select in an international context a single test, including both components mentioned above (open-response and closed-response), unlike what unfortunately happened in AHELO. This decision by the CG, notified to ANVUR in October 212, on the one hand strengthened the position of CAE as the only provider in the world to offer such a test, called CLA+ (because it is derived from the CLA). This is all the more so because the CLA had already been identified by the Working Group on Pre-feasibility as the best international product for open-response testing. And, on the other hand, it meant that ANVUR did not have to prepare a call for tenders, but just enter into a private negotiation with the only existing monopolist, especially since it was planned to not spend more than 2,. Then commenced lengthy negotiations between ANVUR and CAE, both face-to-face and at a distance, leading to a huge reduction of the costs originally requested by CAE and an equally large shift in 19 The scorers, selected by the Universities that participated in the project, are listed and thanked in the preamble to this Report. ANVUR had calculated the total number needed and the percentage for each site according to the proportion of eligible students in each University on the total number of eligible students for all Italy. The scorers were thus calculated at a ratio of about of 1/2 eligible students in each University. 3

31 responsibility for activities necessary for the TECO from the Americans to the Italians. Our colleague and esteemed law expert from the WG, Anna Maria Poggi, worked hard to ensure that ANVUR signed a balanced contract with CAE: not too invasive by the US company, as it wanted in the beginning, and with sufficient provisions, wanted by ANVUR, in terms of control over the product and ownership of the process. CINECA cooperated perfectly in this sense (thanks particularly to Mauro Motta) and designed a redirect computerised process both during test administration and after, which effectively guaranteed this ownership 2. Meanwhile Alfonso Caramazza, Piero Cipollone and Roberto Ricci members of the CG and high-level experts in neuroscience or psychometric assessment through testing, devoted much time and energy to examining several examples of tests produced by CAE, for both the open-response and multiple choice parts, and discussing at length with CAE on modifications and adaptations to the test and in the scoring methodology 21. The contract between ANVUR and CAE was finally signed on 15 February 213 after four months of negotiations on various fronts (see Annex 1). Unlike the original CLA test, the CLA+ in the Italian TECO version consists not only of an open-response part but also includes 2 closed-response questions with 4 response options each (the key and 3 distracters). It lasts 9 minutes in total (6 minutes for the open-response part and 3 minutes for the closed-response questions). The openresponse part is called Performance Task (PT), while the multiple-choice questions are called Selected 2 Notwithstanding, the ANVUR-CAE contract provided for signing a confidentiality agreement (see Annex 11) related to the use of the tests, which left little margin of freedom for ANVUR. An obligation concerning confidential information was established whereby, according to the words of the contract, Confidential Information shall mean any confidential or proprietary information, as determined by CAE, that CAE may disclose to the Consultant, orally or in writing, in connection with Consultant s employment, including, without limitation, any test results and data obtained therewith, trade secrets, methods, software and associated documentation, business plans, source code, inventions, processes, designs, drawings, engineering or hardware configuration information, know-how, or any other proprietary or business information. By way of example, the methods employed to create the CLA tests are considered trade secrets of CAE and must be treated as Confidential Information. Ownership of Work Product All work performed by the Consultant for CAE is owned by CAE and is considered Confidential Information. There shall be no dissemination or publication of any work or information developed during Consultant s employment without the prior written approval of CAE. Permitted Use The Consultant shall use Confidential Information only during Consultant s period of employment and solely for the purpose of providing services to CAE in accordance with the terms of this Agreement, the policies and procedures of CAE, and any employment agreement that may be in effect. The Consultant shall not use any mentally-retained recollections of Confidential Information to copy, reproduce, summarize, disclose, or make use of the contents or substance of Confidential Information. For example, the Consultant shall not use or replicate, in whole or part, any of the methods used to create the CLA tests. Confidential Information shall, as between the Consultant and CAE, remain the property of CAE. Similarly, all those who were able to examine the contents of the CLA+ or the TECO had to sign a Confidentiality Agreement. As it turned out, against every possible expectation, it happened that CAE, and not ANVUR, breached confidentiality for a few hours, without any consequences. Taking into account all the foregoing, ANVUR, with the consent of CAE, is able to publish the entire open-response part of the TECO test and also an excerpt of the closedresponse questions, so that Italian stakeholders may better understand the characteristics of the test (see the Annex to this Report). 21 In addition to providing the test, CAE committed also to the following activities as per the terms of the contract with ANVUR. Training the two Leads of the LSCs, Roberto Ricci and Fiorella Kostoris, and, subsequently, the twelve LSCs appointed in each of the participating Universities. Offering a technical support manual for learning the scoring method of the performance test, which the LSCs used in turn to teach the scoring method to the Italian scorers (see Annexes 12, 13, 14 and 15). Writing a brief introductory text to explain to Italian students and teachers the principles and rationale of the CLA+ test, to be used also in the pre-test phase (see Annexes 16 and 17). Scoring the closedresponse tests (SRQ) received in anonymous form. Collaborating in the first statistical analysis of test results by drafting a specific Item Report with indicators on the statistical reliability of the answers from the sample of students who took the TECO (see Annex 18). 31

32 Response Questions (SRQ). Performing well in the test does not require any specific knowledge in any particular field. Conversely, both in the PT and in the SRQ the student must take account only of the information contained in the documents included in the test, not from any other external sources although everyone must obviously rely on his own so-called personal encyclopaedia The Performance Task (PT) in TECO In the PT module of the TECO, a stimulus, i.e. a fact or an act or circumstance, of a realistic nature, is presented in a central document, which identifies a theme, with a set of additional, sometimes inconsistent pieces of empirical evidence, exhibiting varying degrees of robustness (totalling six documents). Students are encouraged to take an active role in tackling the issue, suggesting a solution or recommending the most appropriate intervention or deciding between several options presenting desirable and less desirable aspects, based on the information provided in the Document Library. There are no right or wrong answers, but only better or worse argued ones, coherent or incoherent ones, solid or weak answers on an empirical level, and answers described with greater or lesser efficacy and appropriateness of language. The Document Library normally includes several sources of information, which may comprise technical reports, data tables, graphs, newspaper articles, memos, e- mails or similar documents. PT questions are intended to test three aspects: a) Analysis and Problem Solving ability (APS), demonstrated by students in interpreting, analysing, and evaluating the quality of the information and data presented to them. They must, among other things, identify ideas or facts in the documents that are relevant to a problem being discussed, present related or conflicting information, detect faults in the logic and/or questionable assumptions, explain where and how it can be determined that the evidence is credible, weak, unreliable, inconsistent or incomplete, and weigh up information from various sources to make a decision or reach a logical conclusion, underpinned by a coherent analysis of the information provided. b) Writing Effectiveness (WE), shown by the students by communicating their arguments in written form. c) Writing Mechanics (WM) used by students, with regard to the basic rules of the language in which they are expressing themselves. Each of the three areas receives a score from to 6: the minimum score in the PT module is therefore while the maximum is 18. However, students who receive an overall PT score of less than 3 are eliminated from the assessment, as it is believed that in such circumstances the level of their engagement with the test is so low that such a case is observationally equivalent to non-participation in the test (see paragraph 3.14). 32

33 The 2 multiple choice questions or Selected Response Questions (SRQ) in TECO The 2 multiple choice questions have the aim to assess a set of competences of different, predominantly scientific-quantitative nature. For these, students must choose the key (the correct answer), discarding the three distracters, on the basis of the information given or inferred from the documentation supplied (this also includes letters, dialogues, tables, photographs, graphs, newspaper articles or similar). SRQ questions are intended to test three aspects: d) Critical Reading ability (CRE) of a short text, usually accompanied by a graph or other simple quantitative analysis instrument. e) Critique an Argument ability (CA) by selecting, for example, the most convincing position from several expressed by different people and explaining why. f) Scientific and Quantitative Reasoning ability (SQR) in the face of information and pieces of evidence of both qualitative and quantitative nature. Each question receives a score if the answer is incorrect or missing, a 1 score if the answer is correct. Therefore the minimum for the SRQ module is and the maximum is Adaptation of the CLA+ and its transformation into TECO It is not enough to translate a test into Italian, it also needs to be localized ( Italianized ) if we want to offer our post-secondary students a set of documents consistent with the culture, history and context of their country, giving them a test which, mutatis mutandis, truly is identical to that sat by their colleagues at American Universities (or other Universities throughout the world, e.g. those of the nine countries participating in the Generic Skills Strand of OECD-AHELO). Adaptation was performed by members of the CG with professionalism, both by examining various open-response and closedresponse test options from those existing at CAE, with a view to choosing the most appropriate ones, and by asking the American producer to make changes necessary for an Italian environment. Therefore, for instance, the CG discarded in the field of open-response tests those for which cultural references were impossible to reproduce in Italian. As an example there was a great PT test where the stimulus (the central document of each open-ended question) illustrated the painting The Fall of Icarus by Bruegel the Elder exhibited in the Museum of Brussels. Of course it was not required that students have knowledge about Flemish painting or the Greek myth, but the accompanying documents (always existing in every PT) were taken from English literature, with poetic passages by Auden and Elliot. These were not only difficult to translate, but deemed impossible to Italianize by the ANVUR experts, as they were unable to find corresponding Italian writers and poets commenting the same episode of the story of humanity. Again, for instance, in the domain of closed-response tests, those where the theme seemed prurient (linked to the sexual behaviour of young people or feminist struggles) were discarded. Likewise, those for which the statistical characteristics of results already obtained in American colleges did not seem sufficiently robust (for example, because the percentage of 33

34 correct answers was too small and, of the few who answered well, the individual correlation with the rest of the correct answers in other parts of the test was too low). Next, the Italian adaptation consisted of a) reducing the number of accompanying documents in the PT test, based on the assumption (which regretfully proved to be founded) that Italian youngsters are not used to fast and accurate reading, and b) limiting the SRQ part to 2 questions instead of 25 as in the equivalent CLA+ test sat in the USA. Thus the TECO as a whole could be kept down to 9 minutes (with 3 minutes for the SRQ part), to avoid too much concentration time on the test 22. Further, the adaptation of the test to Italy consisted of deleting from the SRQ those elements of the questions that could appear too mathematical to our students, based on the hypothesis (this also unfortunately proved correct) of a literary focus mainly still dominant in the culture of Italy. For example, the Committee of the Guarantors decided that the questions could not refer to more than one graph or table per topic, and that there should be no references to statistical indicators (such as the t- statistic), which in Italy are largely unknown in all disciplines other from those closely related to the hard sciences. The Committee of the Guarantors decided it was nevertheless unable/unwilling to change too radically, in transforming the original CLA+ into the TECO, a test that has been validly tested elsewhere in the world. This is both because it considered that the target audience of Italian students (and their families, teachers and society) should begin to adapt to international standards, and because these provide the benchmark of comparison, absolutely essential for us to arrive at proper evaluations of the results observed in Italy. Therefore, the CG and ANVUR concluded that in adapting the test, excessive deviations between the CLA+ and the TECO should be avoided. 3.6 Presentation of the experiment to the stakeholders via seminars The administration of the test on the generic competences of Italian graduating students was preceded by an intense phase of presentation and discussion of the initiative with the twelve Universities involved. The participating Universities organised meetings with ANVUR from the end of November 212 until mid-february There were many participants and high interest from academic and administrative staff, students and sometimes potential employers (thanks to the framework that the twelve Universities gave to the seminars). The presentation of the initiative at local level was a good opportunity to explain the reasons, criteria and methods of the TECO pilot test, as well as to hear the views of stakeholders both internal and external to the Universities. The latter were expressed in 22 In the original CLA+, initially used in the United States, the PT module alone generally lasted 9 minutes, with 9 documents. Subsequently, a PT module lasting 6 minutes and with 6 documents, as in the TECO, was also proposed in the USA within the CLA+. 23 The dates of the seminars were: 29 November 212 at the University of Eastern Piedmont; 1 December 212 at the University of Salento; 11 January 213 at the University of Florence; 18 January 213 at the University of Udine; 21 January 213 at the University of Messina; 23 January 213 at the University of Rome La Sapienza; 24 January 213 at the University of Rome Tor Vergata; 28 January 213 at the University of Milan; 31 January 213 at the University of Bologna; 1 February 212 at the University of Padua; 11 February 212 at the University of Cagliari; 18 February 212 at the University of Naples Federico II. 34

35 numerous interventions, observations, questions, comments, and constructive suggestions except marginal episodes which showed some opposition on the part of the students or teachers 24. Of particular importance were the comments made by those who operate in the job market, and are thus aware of career opportunities for graduates (companies, banking foundations, the General Government, etc.). In any case, much time was set aside for debate, avoiding compressing it with lengthy interventions by the speakers. Among the issues most often the subject of questions, were those on the fallout of the test and on the operational procedures with which actual learning outcomes are assessed. The seminars helped to convey to listeners (also via streaming), together with more information, a certain level of enthusiasm for the initiative, avoiding falling into the trap of the pilot test being seen either as additional bureaucratic burden by the administrative staff, or as dangerous interference by the teaching staff, or as just additional exams by the students. The teachers were made aware of the strategic significance for the University of the outcomes of the experiment, as the results obtained will be able to contribute, among other things, also to a significant improvement of teaching methodologies currently used, following a thorough self-assessment. 3.7 Set-up of the technological platforms and collection of the contextual variables Carrying out the TECO required the use of two technological platforms: those of CINECA and CAE. CINECA began working with ANVUR and with the twelve Universities participating in the pilot test, on the one hand, as well as with CAE, on the other hand, in November 212, because matters needed to be agreed on several fronts. In the first place, CINECA and ANVUR needed to agree on a system for the pre-registration of eligible test candidates, on the form with the questions they would need to fill in, and on how to collect all contextual variables concerning non-sensitive data about students and Universities. In cases where students and/or Universities would fail to provide this data to the Agency in timely and accurate manner, CINECA committed to submit any empirical evidence collected to ANVUR, in the manner and time required. CINECA also had to agree with CAE the process for reproducing the test electronically for test takers, so that, for example, the screens on the PC would be the same as those seen by students tested elsewhere in the world and that testing times would be equally strictly observed (6 minutes for the 24 As an example of the former, some young people intervened to say they are tired that exams never seem to end for them (obviously not understanding ANVUR s intent of kicking the dog and meaning the master ). A case of real antagonism occurred in one University, when some students said: We will boycott and will ask our fellow students to boycott the test because we do not believe that you want to assess our critical analysis ability all you have ever wanted from us is one-track thinking. As an example of the latter, there was criticism of the inappropriateness of assessing teaching quality externally or doing so via a test, and rather about the opportunity to do so with a disciplinary test, or to do it but to make the individual results available to the Universities in a non-anonymous form. On the first points we refer to Kostoris Padoa Schioppa (212) for our replies. On the last one, we wish to officially state that ANVUR agrees that, in the future (once rolled-out), the TECO shall be implemented with full disclosure of results through a coordinated but firm discussion with the Privacy Guarantor, which so far has prevented this, perhaps even resorting if necessary to some changes in legislation. 35

36 PT, 3 for the SQR). CINECA committed to ensuring that the scoring of the PT could be done online with the same characteristics normally used by CAE, but also taking into account certain ANVUR requirements (for example, leaving space also for a short comment in addition to the score), allowing the opportunity to review the judgment after a first formulation, offering the LSC the opportunity to check the scorers' work, and allowing the Leads of the LSCs to monitor, from INVALSI, each individual scorer. CINECA also had to agree with CAE on the methods for redirecting a student about to take the test to their platform, once his/her identity is released to CINECA, made anonymous and ready to pass on to the American platform. It also had to agree on the return stage from CAE to CINECA, upon completion of the 9 minutes testing time, of the complete string of information with the answers given by each tested student. In addition, CINECA had to agree on a series of elements with staff in the data centres and administrative offices of the twelve participating Universities, in order to help them to extract information on students eligible for the test together with their contextual variables. The graduating students who intended to sit the test had to pre-register by completing a CINECA online form. Sometimes this was combined with another online form from their University (Felisatti, 213): the first form was an application to sit a given session of the TECO test, while the second form provided information on the details of the site or times at which the student was admitted to sit the test. Thus, when pre-registering on the CINECA platform, the student provided his/her basic information (see Annexes 19 and 2), starting with name and ID. Later, at the time of testing, he/she would add more information, such as parents profession and level of education, perception of the competences acquired on the course, attendance regularity, as well as a waiver for ANVUR concerning sensitive data. Table 3 on contextual variables shows the different degree of existing information about the students enrolled in the third and fourth year in the twelve universities depending on whether they came to sit the TECO, they pre-registered but did not show up for the test, they did not pre-register despite being eligible, or they turned out to be ineligible. 36

37 TABLE 3: Contextual variables in the 12 participating universities, for eligible and ineligible students Type of existing information per student category: Source Acronym Tested, Pre-registered, Not pre-registered (eligible and ineligible) students S: Student Pre-registered (eligible and ineligible) students U: University Tested students ANS: National Student Register - MIUR (ANS) O: Educational Offer - MIUR (OFF.F) ANV: ANVUR M: MIUR PERSONAL UNIVERSITY BO-PD PRV PRN N PRVA PRE N ID BIRTH DATE S* BIRTHPLACE (MUNICIPALITY) S* GENDER S* MARITAL STATUS S** PROVINCE OF RESIDENCE S* MUNICIPALITY OF RESIDENCE S* YEAR DIPLOMA OBTAINED ANS SCHOOL LOCATION (PROVINCE) ANS SCHOOL LOCATION (REGION) ANS SCHOOL LOCATION (COUNTRY) ANS UNIVERSITY NAME (CITY) ANS COURSE U CLASS ( ) DM 59/99 DM 27/ DISCIPLINARY GROUP ANV MACRO-GROUP MIUR GEOGRAPHIC AREA U GDP GROWTH RATE IN REGION WHERE UNIVERSITY IS LOCATED ISTAT tot tot tot tot tot tot tot tot tot YOUTH UNEMPLOYMENT RATE IN REGION WHERE UNIVERSITY IS LOCATED ENVIRONMENT ISTAT tot tot tot tot tot tot tot tot tot GDP GROWTH RATE IN REGION OF BIRTHPLACE ISTAT tot tot tot tot tot tot tot tot tot FAMILY SOCIAL SUPPORTS FOR STUDYING INDIVIDUAL MERIT EXTERNAL MERIT (*****) Variable Name NUMBER OF INELIGIBLE STUDENTS YOUTH UNEMPLOYMENT RATE IN REGION OF BIRTHPLACE ISTAT tot tot tot tot tot tot tot tot tot NUMBER OF MEMBERS OF HOUSEHOLD S NUMBER OF SIBLINGS WHO ARE STUDENTS S LANGUAGE SPOKEN AT HOME S CITIZENSHIP S* OFF-SITE (distance place of residence - place of study > 2 km) S MEAN TRAVEL TIME UNIVERSITY-RESIDENCE S WORKING STUDENT (as reported by student)**** S WORKING STUDENT AS PER ANS (information from ANS) ANS OWNS PC S OWNS TABLET S OWNS SMARTPHONE S ATTITUDE TO TRAVEL S AVERAGE NUMBER PER YEAR TRIPS OUTSIDE REGION S AVERAGE NUMBER PER YEAR TRIPS ABROAD S FATHER S PROFESSION S FATHER S EMPLOYMENT CONTRACT S FATHER S STUDY QUALIFICATION S MOTHER S PROFESSION S MOTHER S EMPLOYMENT CONTRACT S MOTHER S STUDY QUALIFICATION S SCHOLARSHIP S STUDENT RESIDENCE S MEAL VOUCHERS S STUDENT COLLABORATION CONTRACTS S OTHER S NATIONAL ADMISSION TEST O LOCAL ADMISSION TEST O FOREIGN LANGUAGES KNOWN S NUMBER OF COURSES FOLLOWED IN FOREIGN LANGUAGE, IN ITALY S NUMBER OF COURSES FOLLOWED IN FOREIGN LANGUAGE, ABROAD S NUMBER OF MONTHS ERASMUS OR OTHER PROGRAMME S HIGH SCHOOL DIPLOMA GRADE ANS HIGH SCHOOL TYPE ANS AVERAGE UNIVERSITY GRADES (EXAMS SAT SO FAR) ANS NUMBER OF UNIVERSITY EXAMS SAT SO FAR ANS TOTAL CREDITS ACQUIRED ANS QUALITY OF EDUCATIONAL ENVIRONMENT - VQR R12 ANV tot tot tot tot tot tot tot tot tot SELF-ASSESSMENT OF EDUCATIONAL ENVIRONMENT - EXPECTED COMPETENCES ( SUA FORM) ANV tot tot tot tot tot tot tot tot tot QUALITY OF STUDENT ENVIRONMENT - M INDEX ANV tot tot tot tot tot tot tot tot tot SELF-ASSESSMENT OF STUDENT ENVIRONMENT - ATTENDANCE REPORTED AS REGULAR S SELF-ASSESSMENT OF STUDENT ENVIRONMENT - STUDENT PERCEIVES COMPETENCES ACQUIRED IN UNIVERSITY AS ADEQUATE FOR TECO S (*) Although the source is the students, for the categories N and NN the source for the variables BIRTHDATE, MUNICIPALITY_BIRTHPLACE, GENDER, MARITAL_STATUS, PROVINCE_RESIDENCE, CITIZENSHIP is the University (U) For Bologna and Padua, refer to the note (***). Source NUMBER OF ELIGIBLE STUDENTS (**) Although the source is the students, for the categories N and NN the source for the variable MARITAL_STATUS is the National Student Register (ANS) For Bologna and Padua, refer to the note (***). (***) For the Universities of Bologna and Padua, the source of the data for the categories N and NN (columns BO-PD_N and BO-PD_NN) are the universities themselves, which provided the data in anonymous form. As can be seen, these Universities have provided data on fewer student characteristics, particularly as regards ineligible students. (****) The information can be different from what is found in ANS, since it was provided directly by the student when pre-registering. In this case, by working student we mean a student with any type of occupation (even a precarious one) which is systematic and remunerated. (*****) The variables under External merit exist for different aggregations of University variables. See TABLE 3b for more details on the University contextual variables NPRN NN BO-PD NN 37

38 There are numerous statistical sources for such information: the MIUR Database on the educational offer, the MIUR National Students Register, the students themselves, the Universities, and the National Statistics Institute (ISTAT). Some vectors were then identified, for homogeneous groups of contextual variables. This includes objective individual data, such as those of a demographic nature (e.g. gender, marital status, age, time since diploma obtained), whose influence on the TECO cannot be imagined ex ante. Likewise objective family data (e.g. number of household members), with the same uncertain expected effect on the TECO. Differently for i) social demographic data (e.g. citizenship, language spoken at home, the condition of living off-site or being a working student, parents social or cultural class, ownership of IT equipment, attitude to and frequency of travel), ii) individual meritocratic data (diploma grade and type, University grades, admission test passed at national or local level, knowledge of foreign languages or courses abroad), and iii) collective meritocratic data concerning externalities related to the study and academic environment. For these there are instead expected effects on TECO (considering the plentiful existing studies on the economics of education), which, however, must be empirically checked through analysis of the pilot test results. 3.8 Translation and conciliation of the texts The translation of the PT and SRQ modules of the test was conducted in the first instance independently and excellently by two organisations, that we would now like to thank most sincerely, INVALSI who did it for free (with Maria Alessandra Scalise) and the Company CAPSTAN of Brussels (with Andrea Ferrari), specialising in translation/validation of tests materials in the domain of education. The two translations were then compared and problems of inconsistency or differences between the two versions were resolved through a conciliation process, carried out by CAPSTAN, under the supervision of ANVUR and with the approval of CAE. The entire complex operation took about a month. In addition to the test itself, various other documents required for test administration, explanation, scoring, etc., were also translated, for example, the Scoring Manual (see Annex 12). This Report is also translated into English Focus group and cognitive laboratory at the University of Camerino The actual administration of the TECO was preceded by a pre-test phase carried out at the beginning of the spring (5 April 213) at the University of Camerino. Forty-four students, evenly distributed by type of study course, attended the focus group. This amounts to 21.78% of the 22 eligible students, a 25 ANVUR is grateful to the two financial sponsors Caripuglia and the Friuli-Venezia Giulia Region University of Salento which, via the University of Salento and Udine, should cover all translation costs. 38

39 percentage, as will be seen, only a little lower than the mean for the twelve Universities participating in the pilot test. Forty-two students completed the test, while two withdrew (see Annex 26). The focus group was used, as always in these circumstances, to detect the presence of printing errors in the test and assess the appropriateness of the translation from English for the purpose of getting a good understanding of any possible residual problems, as well as to fill some of the gaps concerning background information, sometimes asked to the students themselves for the purpose of better identifying the contextual variables that characterise them. These evaluations were carried out with the cognitive laboratory technique, on the basis of a guide produced by CAE (see Annex 2). Given that how students reason when answering the tests and their thought processes cannot be directly observed, inferences need to be made on their verbal responses. The laboratory therefore uses the think aloud method: the participating young people are asked to think aloud while they solve the problem posed by the question and no one, at this stage, interrupts them with explanatory or corrective interventions (some questions, intended to collect further elements, are posed at the end of the test session). The cognitive laboratory run by ANVUR had the special objective of checking that the Italian translation did not alter the original test constructs, that questions were interpreted by our students with the original meaning they had in English, that they were not, in general, more difficult to read or understand than if they had been written for Italians from the outset. The pre-test showed reading without difficulty by all and the same for Italian students with respect to what has emerged from similar tests in other countries of the world, even very different ones like the nine participants in the AHELO feasibility study on generic competences. The pre-test was in any case quite useful. Thanks to the contribution of one student, it brought to light a couple of errors in the content of the documents that accompany the stimulus in the open-ended part, and it was instrumental for suggesting small lexical improvements for easier and more unambiguous understanding of the texts. 3.1 Validation of the translation after the focus group The errors and minor translation issues brought to light by the focus group at the University of Camerino were then corrected and resolved, thus leading to a final, validated translated test. The TECO package also included forms (see Annex 19) to be filled in by students who pre-registered and then for those who came to sit the test. Students were asked to provide personal information, as already indicated in Table 3, including: demographic data on the student and on the composition of the household, family socio-economic status, off-site or working status, any form of support for studying received, individual data of meritocratic nature (diploma and University grades, admission test passed at national or local level, their perception of whether they have acquired competences in their course 39

40 of study, attendance regularity). Also, all pre-registered candidates who came to the test had to sign a waiver for the purpose of using their data, as required by the Privacy Guarantor Test Administration The test was administered in the twelve selected Universities over the period from 27 May to 4 July 213: the dates vary from one University to another, according to choices made by the Universities themselves (see Annex 21). In any case, continuity in test administration over a defined period was ensured, so as to ensure homogenous data collection. In some cases, as in the University of Udine, the test sessions were limited to a 3-day testing window (from 17 to 2 June); in others, such as the mega Universities of Bologna and Milan, the sessions were extended over a testing window of more than a month (from 3 June to 4 July). With the data available on Italian Universities as a whole and on the twelve participating Universities (Tables 4 and 6), it was known that students eligible to sit the TECO would be just under 2% of all students enrolled in the third and fourth years of all study courses excluding those for the health professions, i.e. a population in the academic year of 21,872 persons. For the abovementioned reasons, it was expected that no more than 1-11, students would take the test. As it turned out, 14,97 students pre-registered for the test a number which includes many ineligible extraneous students, who were not admitted to the test whereas of the eligible and pre-registered students, only about 5,9 students actually showed up to sit the test (Table 7). 4

41 41

42 Each University prepared a participation certificate to be given to students upon completing the test, drawn up according to an agreed uniform template (see Annex 22). The students were also offered, as already mentioned, the possibility of obtaining from ANVUR, upon individual request, a certificate with the test result. About three months down the line, 3/5 of the tested graduating students have already asked to know their results (see Annexes 23, 24, 25). All the TECO test sessions were administered online in a supervised environment. In any case, it is not easy to cheat (as might happen in a test in school) on the PT part of the test, by its very nature. Likewise for the SRQ part, as the 2 questions were randomly distributed to students, so that the first question for one student could match his/her neighbour's last one. It was therefore not considered necessary to apply methods to estimate any cheating effect which does nevertheless exist where the above precautions are lacking, for example in a number of paper-and-pencil tests organised by INVALSI. The summer period was chosen to administer the test, leading to some complaints by the Universities concerning e.g. the reduced participation of off-site students with respect to pre-registration, owing to the temporary absence of young people from the city where the University is located (lessons are over for the academic year and the TECO test session does not coincide with the exams session). However, on the contrary, other students (presumably those not off-site) justified their non-participation in the 42

43 test precisely with the opposite reason, i.e. with the excessive overlap of the TECO period with that of the examinations. Both positions are in fact weak, judging from Table 8. It is observed that off-site students are often more present among those who came to sit the test versus those who pre-registered but who did not then show up for the test. On-site students, on the other hand, were more often frequent among those who did not show up than among those who did come to sit the test, proving that non-participation is only weakly dependent on being off-site. There are about the same (or slightly less) off-site students among those tested than among those eligible and pre-registered who did not come to sit the test 26 (except at the University of Rome Tor Vergata, where off-site students were much more numerous among those who in the end did not sit the test, and except in Bologna and Padua, where they were instead fewer). These and other factors potentially influenced the characteristics of the sample of pre-registered students and of those who actually sat the test with respect to the set of all eligible students, creating a possible distortion due to self-selection (positive if the worst students self-exclude themselves, 26 It is certainly true, however, that, if the TECO test had taken place on 1 November instead of 1 April 213, the number of eligible students would have increased to 16,637, excluding those (who number 12,177 a sizeable figure) who do not comply with the requirements for basic or characterising credits on 1 April, but who graduated in any case between 1 April and 1 November 213 (see Table 1.1 shown later). 43

44 negative if the opposite happens). The available data allow the problem to be assessed with some approximation, but do not allow the self-selection bias to be eliminated Test administration management by CINECA and CAE, up to the release of individual results Two technology platforms were used to manage the TECO, CINECA s and CAE s. The former was used to collect and score test answers on the open-response part, as well as to process all the data characterising (graduating) students in the twelve participating Universities, while the latter was used to process the multiple choice questions and the related scoring. Once the PT module was scored in Italy, CINECA sent the scoring data back to CAE after making it anonymous also as regards the scorers 27, but the last transfer from CAE to CINECA took place after the superscoring by CAE, which put together the scoring of the PT module and that of the SRQ module for each graduating student. As regards privacy regulations, the constraints that emerged made it so that the Universities were not authorised by the Guarantor to receive information on the (non anonymous) individual TECO performance of their graduating students. This meant they were precluded from using this type of parameter for any form of incentive or reward for better-performing students. In view of this, ANVUR decided, via CINECA, to provide to those who sat the test, upon request, a certificate with their results for each of the six aspects of the assessment, both in absolute terms and in relation to various benchmarks. Starting from 11 March, ANVUR deliberated (on 17 February 214) that, in any case, for transparency purposes, Universities could access all the anonymous individual data not only of their own tested graduating students, but also of those in all twelve Universities participating in the pilot test Training of Lead Scorers (LSCs) and Scorers (SCs), and scoring of the openresponse test (PT) The process of scoring the open-response component of the TECO was a delicate matter, requiring adequate preparation from a methodological point of view and specific training of the Italian teachers who would carry out this task. Scoring the 2 closed-response questions (the SRQ component) was instead carried out directly by CAE, who received the answers in strictly anonymous form: SRQ does not require any discretional or judgemental assessment, so the scores are assigned by a computer. Scorer training for the PT component of the TECO took quite some time. For most or even all the distinguished teachers who underwent this training, it was their first experience of this kind, never 27 In the test administration phase, the transfer from the CINECA platform to the CAE platform took place via an internal redirecting operation by CINECA, without the testees being interrupted in the completion of the TECO, indeed without them even noticing it. After the test was finished, a reverse transfer from CAE to CINECA for the PT part took place. 44

45 previously received in their study and work life. Such experience would always be desirable: while competence and intellectual honesty are a necessary condition for a fair assessment of the student being observed, they are hardly sufficient to guarantee a neutral rating, in the sense of being independent of the order in which the test or exams are rated, and also of being replicable by other members of the same scoring committee. Two people (Fiorella Kostoris and Roberto Ricci) had two days of online training and two days of faceto-face training, courtesy of CAE expert Doris Zahner (whom we would like to thank for her great helpfulness and the contribution of very high scientific value that she offered in this training and even more so later, in the statistical analysis of the results, see Annex 18 and Zahner, 214). Roberto Ricci, a psychometrician from INVALSI then performed the role of Leader of the twelve LSCs. The scorer training session, to which only the twelve LSCs from the participating Universities were invited, was held at ANVUR on 24 and 25 June 213 and led by Doris Zahner. The LSCs, after going through this training from the American company, in turn proceeded to train the scorers (SC) at their respective Universities. The training for scoring the PT component, carried out in Italy, was accompanied by content analysis tables in support of the scoring, specifically drafted with the help of a CAE Manual (see Annex 12) and a further Scorer s Guide, skilfully written by Fabio Vendruscolo, ICP and LSC of the University of Udine (Vendruscolo, 213; see also Annexes 13, 14, and 15), providing detailed instructions for each of the areas covered by the test (APS, WE, and WM). CAE itself rated this Guide as excellent. A subset of 2% of the questions scored by each Italian SC was also scored by another SC, and all scorers were assigned questions to be scored anonymously and randomly, i.e. without knowing who they were rating or whether it was their own or someone else s student. For each double-scored answer, if the scoring by the two SCs involved was not consistent, i.e. if the difference between the scores assigned by the two scorers was greater than 2 for each of the three areas of assessment, INVALSI intervened to carefully monitor the situation. In fact, the quality control process over the scoring, entrusted to INVALSI, was made easier by the fact that 2% of the answers were scored by two independent scorers selected at random and thus constituting variable pairs. In this phase, SCs that made rating decisions that had no solid foundation were immediately identified, because their ratings emerged as outliers with respect to those formulated by all other scorers. One evidence of the successful scoring operation carried out in Italy on the TECO is the very small number of outliers among scorers: in all, 3 out of the 11 persons involved in scoring the test. The ratings carried out by these three scorers of less high quality were subsequently checked and revised by INVALSI. In this respect, we would like to warmly thank Cristina Stringher and her monitoring group. Each SC in each University could do the scoring on his/her computer, online, directly using the CINECA platform. Each SC could be (and had to be) monitored directly by his/her LSC. The scoring task was 45

46 completed within 31 August , so that the Lead LSC, Roberto Ricci, with his INVALSI group, could review the ratings assigned by the SCs as needed. So: - The final score in the PT module (minimum, maximum 18) was obtained by combining the separate ratings provided by two independent scorers on the three different areas (problem analysis and solving, writing effectiveness, writing mechanics), after reviewing and validating these ratings. - The score in the SRQ module (minimum, maximum 2) is simply the number of correct answers in the 2 closed-response questions. Lastly, the overall TECO result i.e. scaled score for each student to be distinguished from the raw score talked about so far was calculated as the arithmetic mean of appropriate transformations of the scores designed to change them (respectively in the intervals 1-6 and -2) into homogenous scales (see Annex 18 and Zahner, 214). CAE's superscoring work, with the production of the TECO result was completed as expected on 3 September 213, so that it is now available for each tested student for each of the six components of the test Data checking and cleaning Checking and cleaning the data with the TECO result was carried out by ANVUR in collaboration with CAE (see Annex 18) and with CINECA, also making use of information offered by the twelve participating Universities. The main effects of this cleaning are described in Table 7. For example, the test was cancelled for 45 students who sat the test but achieved a PT score of less than 3, a clear sign of their lack of engagement, so that they are considered comparable to pre-registered eligible students who did not sit the test. On the other hand, data checking and cleaning revealed that, in very few cases, Universities allowed some students who did not meet the requirements to sit the test. Considering that once it's done, it s done, all the more if done by others, ANVUR decided to count these students as eligible for all effects, as if the Universities could never be mistaken and therefore had possibly been mistaken previously, when sending in their lists with the identification of requirements for eligible and ineligible students. Lastly, data checking and cleaning brought to light the case not only of a couple of hundred students from the third and fourth year (excluding courses for the health professions), who pre-registered for the test while not meeting the requirements, but also of 5,524 extraneous students (enrolled in years other than the third and fourth year, or enrolled in courses for the health professions, or ineligible for other reasons) who pre-registered for the test, obviously in eagerness to put themselves to a test. If 28 The scoring, being double-blind, could not begin until the last University in order of time had finished the test sessions. In total, it lasted less than 2 months. 46

47 the number of pre-registered students can be a proxy for the TECO s approval rating, we are pleased to note that this number reached as high as 14,97 students, equal to more than 2/3 of those eligible for the test whereas those who actually sat the test were close to 5, Public presentation of the outcomes of the experiment During the 18 months of the pilot test phase there were numerous contacts (face-to-face, at a distance, by letter, etc.) between the various collaborators in the TECO project, Italians and Americans, computer technicians and translators, persons responsible for central or local governance, etc. but the presentation of the results to the external public began only in mid-december 213. First of all ANVUR informed and consulted with the then Minister of Education, Maria Chiara Carrozza, receiving her support and intention to institutionalize, past the experimental phase, the practice of assessing the generic competences attained by Italian graduating students. Encouraged by this engagement expressed from the top, which ANVUR trusts will be confirmed also with Minister Stefania Giannini, new head of the Ministry, the Agency, after debating within the Board of Directors and with the expert Emanuela Reale about the different outcomes of the pilot test, began to disclose the major aspects in a first seminar given in English at the University of Bolzano (see Kostoris Padoa Schioppa, 214) and on its website. The elements emerging from the TECO pilot test and the first simple correlation and multiple regression analyses (mostly the work of the econometrician Franco Peracchi, whom we would like to thank warmly here) were revealed in greater detail in January and February 214 to the majority of speakers expected at the Conference on 11 March. Lastly, just prior to this Conference, on 3 March 214 the ANVUR Advisory Committee received a Report on the principal outcomes from TECO (see paragraph 4.15). 47

48 4. Main facts emerging from the experiment 4.1 The regularity index, R, in University studies The Italian University system has many known problems, as shown also by international comparison: fewer young people coming out of high school enrol in University, compared to their peers in OECD countries; too many drop out (and not only in the first few years); too many graduate late with respect to the normal duration of studies; and after graduating a majority remains for a long time in temporary employment or unemployed (see also ANVUR, 214A). It also suffers from a problem that has so far been little known even to the Universities, as shown by the excerpt of their self-assessments in Table 9.1: the study career of those who graduate, even of those who graduate on time, is irregular (according to meaning that this experiment gives to the word). In the twelve Universities of the experiment only 14 19% of the students of the third and fourth year of a three-year cycle (depending on whether you look at the problem before or after the summer exams session) complete all of the basic and characterising study credits required by their study course by the end of it. Only about 18-21% of the students of the third and fourth year of the threeyear first-cycle courses and single-cycle master courses are in a regular situation. Therefore, in our terms, there are few graduating or eligible students, i.e. students who are entitled to sit the TECO test. As a result, it happens (and it is a mixed blessing) that as much as nearly 2/3 of graduates within the third year of the three-year course (bureaucratically and officially defined as regular students ) achieve their University degree without having completed the basic and characterising courses since at least one semester (see Table 1.1). 48

49 49

50 Table 2.1 shows that the percentage of regular students enrolled in the third and fourth year (regularity index, R) ranges very widely across Disciplinary groups. The best are those who must pass a national admission test (Medicine, Architecture, Veterinary Medicine and Dentistry) or a local admission test with 1% of entrants tested (Psychology) or some with a local admission test for a majority of entrants (e.g. Pharmacy). However, two Disciplinary groups where there is no admission test in any of the Universities of the pilot are also in an excellent position (Philosophy and Law). In the breakdown by Universities, the highest regularities are those of Rome La Sapienza and Eastern Piedmont (see Table 2.4) Note that all of the graphs where is set as the origin (e.g. those related to Tables 2.1 and 2.4) show the distances from of respectively Disciplinary groups and Universities in terms of R and P i.e. of the regularity index and participation index of eligible students thus indicating the differences compared to the mean for the sample of the twelve participating Universities. 5

51 4.2 The TECO participation index, P Table 2.1 and following are also interesting to examine because they illustrate the participation index, P, of graduating students who came to sit the TECO, next to the regularity index, R. On average, about 27% of eligible students attended the test voluntarily (just over half of those desired/expected from the twelve Universities). Hence the percentage (Q) of the students of the third and fourth years excluding health professions whose test results we know is a mere 5%. Another potentially more serious problem is related to the participation index shown in Tables 2.1 and 2.4 respectively. This index varies greatly between Disciplinary groups and between Universities: it is almost twice in 51

52 respect of for Mathematics, Physics and Statistics, while it is 1 points lower than the Italian mean in the Art Group, followed by Psychology, Dentistry, Medicine and Languages. In general, as Table 2.3 shows in summary form, the P index is high only in the Scientific Macro-group, while the R index is high only in the Health Macro-group and, to a much more limited extent, in the Social Sciences Macro-group. The Humanities Macro-group does not exhibit a good level of regularity, R, nor much desire to sit the test, P. It is thus in the south-west quadrant, where unfortunately the Universities of Bologna, Rome Tor Vergata and Naples are also found, whereas the Universities of Udine and Eastern Piedmont are in the north-east quadrant (63-64% of eligible students sat the test, exceeding by far, at least on average, the 5% target). The fact that the participation index, P, is low on average is not of particular concern, because in a following phase of TECO, when the administration of the test on generic competences will presumably become a requirement for students who graduate on time, the P index will rise drastically to almost 1. Furthermore, the main problem caused by not only low but also much differentiated participation rates will become insignificant. When this happens, as in the TECO pilot phase, it is difficult when first analysing the data, as we are doing in this Report, to adequately identify and especially correct the self-selection bias. If this bias is positive, as we will strive to show later, i.e. if the tested students are probably better than the other eligible students who did not come to sit the test, it becomes difficult to 52

53 make certain assertions on the basis of the data observed in the TECO either raw data or data in which other contextual conditions are filtered out through appropriate multiple regressions. For instance, the University of Bologna had greater apparent success in the TECO than Eastern Piedmont. This empirical evidence could mean a higher level of learning outcomes for Bologna, or it could be due to the self-selection of students participating in the test: only 13.91% of the Bologna graduating students came to sit the test, compared with 63.4% for Eastern Piedmont. Rather, it would seem easier to conclude, on the basis of the information on the TECO and on the type of self-selection bias discussed later on in this Report, that the University of Udine probably performed better than Eastern Piedmont and than the five Universities with levels of tested generic competences that are higher than the Italian mean (see Tables 4.11 and 7.1 shown below). Likewise, that the University of Bologna presumably performed better in the TECO than the University of Naples, all with a virtually identical low test participation index. 4.3 TECO passes the feasibility test in Italy Before making such evaluations, it is worth noting that the TECO certainly seems to have made the grade as regards feasibility in Italy. This was the main objective of the pilot phase and it was achieved, as illustrated with a wealth of detail also in the CAE's Item Analysis Report (see Annex 18). We only need to look at Tables as evidence of this. The density distribution approximates a normal distribution with a mean of 1 and a standard deviation of 2. An examination of the frequency distributions of the scores for the two test modules shows some left asymmetry in the PT component, some in the opposite direction in the SRQ component, and a significant difference between males and females (to the detriment of the latter), in particular in the most scientific-quantitative SQR and CRE aspects, and therefore in SRQ (see Tables 3.6 and 3.7). On its own, this Gaussian-type distribution function suggests that the TECO passes the feasibility test in Italy. 53

54 54

55 55

56 56

57 57

58 4.4 Comparability of TECO results and scores between Italian graduating students and similar student populations in the rest of the world In addition, it should be pointed out (Table 3.8) that in the twin CLA+ test, given to 4,38 graduating students of US colleges, the results are virtually identical to our own, for both mean and quartiles (except in the highest quartile, which is higher in the US), illustrating superior writing effectiveness and technique in young Italians, as well as greater ability to argue and in critical reading, but lower scientific-quantitative reasoning quality. There is a possibility to validly compare graduating students from the USA, from Italy and from various countries in the world, including some that are very different from one another (Table 3.9). This is because the generic competences measured by the TECO pilot test and by the OECD feasibility study (AHELO, 213) are all assessed with the same CLA-type open-ended response test 3. 3 For reminder, the CLA test, unlike the CLA+, comprises only the performance task, PT, and not the SRQ as well, as in the CLA+. 58

59 4.5 The specifically Italian problem of the two cultures The only aspect that gives cause for concern in the results of the Italian TECO test, compared with the identical American CLA+ test, is clearly shown in the lower part of Table 3.8. The correlation, individual by individual, between the scores obtained in the literary part of the test (PT) and the scientific-quantitative part (SRQ and particularly SQR) in Italy is half that in the United States. This is a first sign of the so-called two cultures existing in our country. Regardless of the average level of competences acquired at the end of University studies by our students, they normally show logic competences that are much more dissociated between the humanistic and scientific domains versus what is observed elsewhere in the world. An esteemed mathematician colleague suggests calling this the Croce-Gentile effect (with uppercase C and G, or perhaps lowercase letter should be used?), indicating that the problem stems from way back, from Italian history and cultural roots, and it is certainly not attributable to the faults of our University system. Our Universities, however, could do more to compensate for the disparity in our students logic competences, nowhere else observed. It would help greatly if Italian Universities were to apply an enlightened Decree from the Ministry for Education, Universities and Research (Ministerial Decree of 22 October 24, No. 27), which is 1 years old but has so far largely remained unapplied. The Decree concerns "course admission requirements", and reads as follows in Art. 6, Par. 1: "To be admitted to a degree course the University's educational regulations require the possession or the acquisition of suitable initial preparation. To this end, the same educational regulations define the knowledge required for admission and determine the assessment procedures, including at the term of 59

60 preparatory learning activities. If the assessment shows an unsatisfactory level, specific additional educational obligations must be met in the first year of the course. These additional educational obligations are also assigned to students of courses for which there is an admission test, and who were admitted but with a lower grade than a predetermined minimum grade. It is clear that, if this regulation were made operational, the assessment of requisite competences would be done with some kind of instrument such as the TECO, and the assessment would be followed up in the first academic year by the obligation to pass any educational debits, for example through forms of mathematical zeroing for those with a more classical training and literary zeroing for those who have instead a more technical-scientific training. The problem of the two cultures becomes even more noticeable when analysing the TECO outcomes by Disciplinary group (Table 4.1) or by University (Table 4.11). Without prejudice to the abovementioned disclaimer as regards the self-selection bias, the best results in the test are obtained in the Medicine group, followed closely by Mathematics-Physics-Statistics and Psychology, where, however, the P index for the participation of the second group is 47%, compared to 17-19% for the other two. The difference versus the national TECO mean is significant for these three groups only 31. The entry selection mechanism is somehow related to this result, with national admission tests (Medicine) or local admission tests extended to 1% of young people (Psychology), or very high self-selection (Mathematics-Physics -Statistics), as evidenced by high grades in the school leaving diploma of those who decide for this study field, known to be stingy when awarding grades (Table 4.1, and Table shown below). 31 The Political Science Group is also added to the multiple regression shown in Table 7.1 with a more or less weak significance depending of the regressors used. 6

61 61

62 There are six groups for which the TECO scores are significantly below the mean of and, unfortunately, the minimum is reached in the Education group. Disciplines of high importance in Italy such as Philosophy, History, Law, and Literature in the humanities-social sciences field or Biology and Engineering in the scientific field seem to exceed the national mean and/or median, but not significantly so. Therefore overall (see Table 4.8) only the Health Macro-group is solidly above the national mean and median, while Humanities unfortunately is solidly below. 62

63 The analysis of the two cultures continues by examining Table 4.6. Observation of how the two parts of the test went, the open-ended, more literary part, and the close-ended, more scientific and quantitative part, shows that on average the results correlate well in the Disciplinary groups, with a correlation index of.61. Medicine, Mathematics-Physics-Statistics and Psychology are on average stronger than the others in both aspects, while Education and Sociology are on average weaker in both components. However, while both parts of the test are well harmonized for Psychology students (in the sense that the differences in the two test results are not significant), for those in Medicine and Mathematics-Physics-Statistics there is a clear and strong difference between them, with a prevalence for scientific-quantitative logic. Unsurprisingly, the same is true in the Engineering, Architecture, and Chemistry Groups. On the contrary, in the humanities, the Philosophy and History groups who perform better in the TECO, surpassing (but barely) the national mean and median show a balance on average between the two components PT and SRQ, which instead is not seen in the Arts and Law groups for which the performance in the first part is significantly higher than in the second. Unfortunately, this is the case also for the Disciplinary groups with below average success in the TECO, starting with the Education group. 63

64 Lastly, the analysis of the two cultures is concluded with extreme clarity in Table 4.7. In the graph, the dotted interpolation line shows the mean correlation between PT and SRQ scores described above. For each Disciplinary group, a continuous light grey line shows the correlation at individual level between the two components of the test. As can be seen from the gradients of all these lines, the individual correlation is very low almost everywhere, as the afore-mentioned comparison between Italy and the United States suggested. It is tempting to state, put simply, that Italian graduating students who perform well either know how to write or how to count; and those who have acquired few competences, generally neither know how to write nor how to count in any case they can do one of the two either much worse or much better. 64

65 Tables 4.12 and 4.13 propose a further analysis of the two cultures displaying the data for the Universities. Also in this case the mean correlation between PT and SRQ is very strong (.93), while that at individual level is very weak. It should be noted, keeping in mind the already mentioned caveats, that the University of Cagliari appears to be relatively successful: on average it performs better than the Centre-South (see Table 4.14) and not significantly below the national mean (see Table 4.11 previously shown). Again put simply, and ignoring the self-selection problem, it could be said that when it comes to generic competences the South begins in Rome, but does not include Cagliari. 65

66 66

67 4.6 The top performers From what has been already said about results above the median (those marked with a + sign), it can be understood that the same segments exhibiting these results include those that would be entitled to a super bonus award, if the rules described in the paper by Fiorella Kostoris Padoa Schioppa (212) were followed (see Table 5.5 and following in part 7: Other Tables). The requirement to be defined top and high performers is even stricter: from Table 5.1 it is clear that the highest proportion of top performers (students with a test result above the national mean of the 1th decile) is found in Mathematics-Physics-Statistics (males and females with 7.73%) and even only among females (with 7.9%), while the highest proportion of high performers (students with a test result higher than the mean for the fourth quartile) is found in Medicine (males and females with 18.32%); the proportion is higher for males in this group (2.74%), whereas female high performers are more frequent in the Mathematics-Physics-Statistics group, where they even outperform the males. 67

68 TAB 5.1: Top, high e low performers per Disciplinary groups, broken down by Gender Disciplinary groups (Macro-group) M + F F M M + F F M M + F F M mat.fis.stat (SC)(+)(*) 141,43 7,73 7,9 8,1 15,46 16,31 14,98 6,7 7,9 6,48 med (SAN)(+)(*) 172,25 7,12 6,83 7,45 18,32 16,1 2,74 3,82 3,9 3,72 sto (H)(+)(*) 111,2 7,2, 1,81 1,53 5, 13,51 1,53 1, 1,81 bio (SC)(*) 16,43 6,64 5,92 8,5 11,33 12,43 9,2 8,98 6,51 13,79 filo (H)(+)(*) 118,2 4,63 4,23 5,41 7,41 8,45 5,41 7,41 4,23 13,51 soc (SOC) 958,4 3,8 4,17, 7,59 8,33, 13,92 15,28, giu (SOC)(+)(*) 19,73 3,66 2,78 5,37 9,27 6,6 14,43 9,4 9,3 9,6 ing (SC)(*) 11,2 3,2 3,94 2,68 7,99 8,66 7,74 7,78 1,24 6,85 polit (SOC)(+)(*) 16,18 2,99,88 5,68 9,95 9,73 1,23 1,95 11,5 1,23 arch (SC)(*) 15,94 2,94 2,94 2,94 7,35 7,6 7,84 4,41 4,12 4,9 agr.al (SC) 984,1 2,88 3,51 2,44 6,47 3,51 8,54 12,95 7,2 17,7 cult (H) 977,99 2,82 1,82 6,25 3,52 2,73 6,25 8,45 8,18 9,38 econ (SOC) 991,15 2,8 2,45 3,18 7,74 5,71 1, 11,18 14,29 7,73 farm (SAN) 975,45 2,79 1,74 5,66 7,61 6,25 11,32 12,69 11,11 16,98 lett (H)(*) 112,51 2,63 1,42 6,12 7,89 7,9 1,2 6,32 7,8 2,4 ling (H) 985,38 2,6 2,65 2,38 6,6 5,82 7,14 8,66 8,99 7,14 odon (SAN)(+)(*) 115,3 2,27 6,67, 11,36 13,33 1,34 11,36 13,33 1,34 vet (SAN)(+)(*) 14,11 2,15, 7,69 1,75 11,94 7,69 9,68 7,46 15,38 psic (SOC)(+)(*) 129,75 2,9 2,52, 8,9 9,43 6,25 2,9 2,52, chim (SC) 995,45 1,89, 3,28 5,66 2,22 8,2 8,49 13,33 4,92 terr (SC) 935,7 1,53 1,97 1,6 4,35 4,93 3,72 2,2 17,73 22,87 comun (SOC) 977,81 1,53 2,53, 6,11 5,6 7,69 12,98 11,39 15,38 form (H) 93,28,78,81, 2,34 2,42, 21,88 22,58, art (H) 965,3,,, 3,13 4,17, 15,63 18,75 6,25 geo (SOC) 933,96,,, 3,77 5,13, 2,75 2,51 21,43 999,53 3,55 2,91 4,5 8,85 7,69 1,55 9,81 9,93 9,62 (+): Disciplinary groups with a TECO median higher than the TECO median (*): Disciplinary groups with a TECO mean higher than the TECO mean Top performer: student with a test result above the national average of the 1th decile (> 127,77) High performer: student with a test result above the national average of the 4th quartile (> 1196,71) Low performer: student with a test result below the national average of the 1st quartile (< 793,24) (**): Percentages calculated on respective total students (M+F, F, M). Upward and downward arrows indicate, respectively, the best and the worst percentage in the column. Source: See TAB A-5.1 TECO % Top performers (**) % High performers (**) % Low performers (**) 68

69 4.7 Simple and multiple correlations between TECO results and contextual variables We now focus on the contextual variables that most seem to influence results on the TECO (the quotation marks are purposeful, as this is not a matter of causation). A summary of results in terms of simple correlations is presented in Table 6.1. Another one in terms of multiple correlations (a work by Peracchi, 214) is set out in Table 7.1. The two types of evidence, where the contextual variables considered are identical 32, are always consistent even if sometimes the simple correlation appears stronger (or weaker) given the multicollinearity between various regressors (for example between diploma grades and the professional position of the parents, both of which influence the TECO) and obviously it weakens (or becomes stronger) under the all other things being equal condition adopted in the estimation through multiple regressions. There is, thus, a systematic downwards relationship between the TECO result and the variables age, female gender (versus male) and residence outside the region of the University's location, as well as an upwards relationship relative to the variables time since diploma obtained, coming from a classical studies high school (compared to other types of high schools), mean diploma and University grades, Italian citizenship and Italian spoken at home (versus non-italian citizenship and language). From the single or multiple correlation analysis, it can be seen that many of the variables considered have different effects on different percentiles of the distribution of the scores. The negative coefficient associated with age is always statistically significant, but tends to become weaker when rising through the percentiles. The negative coefficient associated with the indicator of female gender tends instead to grow stronger and to become statistically more significant when rising through the percentiles. On the contrary, the negative coefficient associated with coming from a technical or professional institute becomes weaker in the higher percentiles, for which in general, the effect of the type of high school attended is smaller. The negative coefficient associated with distance from the University site tends to behave in a very similar manner, i.e. it is quite significant for the lower percentiles, but weakens, ceasing to be statistically significant, in the higher ones. The fixed negative effects of the Disciplinary groups Education and Art, and the positive one of Mathematics-Physics-Statistics, are particularly pronounced in the higher percentiles. 32 One example of variables that do not match, in the two types of analysis, is the case of marital status, examined only from the indicator of simple correlation: table 6.1 shows that being unmarried always improves the TECO results compared to being married, at least in the North and Centre. 69

70 PERSONAL DATA UNIVERSITY DATA NORTH CENTRE SOUTH AGE (-) hardly significant (-) hardly significant (-) hardly significant (-) hardly significant GENDER not significant M > F M > F M > F MARITAL STATUS unmarried > married unmarried > married nd unmarried > married REGION OF RESIDENCE not significant residence in the Region > residence outside the Region not significant residence in the Region > residence outside the Region TIME SINCE DIPLOMA OBTAINED (-) hardly significant (-) hardly significant (-) hardly significant (-) hardly significant UNIVERSITY NAME (CITY) MI, PD, UD > PO BO, FI > RM2 > RM1 CA > NA, LE, ME MI, PD, UD, BO, FI > PO, RM2, CA > RM1, NA, LE, ME arch, bio, farm, giu, mat.fis.stat, med, mat.fis., stat, med, psic > arch, ing, giu, mat.fis.stat, med, psic > arch, art, bio, DISCIPLINARY GROUP** odon, psic, sto, vet > agr.al, chim, agr.al, art bio, chim, comun, cult, econ, med > ing, bio, ling > giu, econ, polit, chim, comun, cult, econ, filo, giu, ing, comun, cult, econ, filo, ing, lett, ling, filo, lett, polit, soc > farm, ling, terr, farm, terr lett, ling, odon, polit, sto, vet > agr.al, polit, terr > form form farm, form, geo, soc, terr MACRO-GROUP SAN, SC, SOC > H SAN > SC, SOC > H SAN = SC = SOC = H SAN > SOC, SC > H NORTH > CENTRE > SOUTH; CENTRE-NORTH > CENTRE-SOUTH NUMBER OF MEMBERS OF HOUSEHOLD not significant not significant not significant up to three > more than three FAMILY DATA NUMBER OF SIBLINGS WHO ARE STUDENTS not significant not significant not significant not significant LANGUAGE SPOKEN AT HOME Italian > non-italian Italian > non-italian not significant Italian > non-italian CITIZENSHIP Italian > non-italian Italian > non-italian Italian > non-italian Italian > non-italian OFF-SITE (distance place of residence - place of study > 2 km) not significant not significant not off-site > off-site not significant MEAN TRAVEL TIME UNIVERSITYup RESIDENCE (in minutes) to 15, 16-9 > over 9 not significant up to 15, over 9 > 16-9 up to 15 > 16-9 > over 9 WORKING STUDENT non working student > working student non working student > working student not significant non working student > working student USE OF TECHNOLOGICAL DEVICES two or more > none or one two or more > none or one two or more > none or one two or more > none or one MEAN NUMBER OF TRIPS OUTSIDE REGION PER YEAR at least one > none at least one > none at least one > none at least one > none MEAN NUMBER OF TRIPS ABROAD PER at least one > none at least one > none at least one > none at least one > none YEAR SOCIAL DATA managerial/professional or whitecollar managerial/professional or white- managerial/professional or white- managerial/professional or white- FATHER S PROFESSION worker > labourer or unemployed, no father collar worker > labourer or unemployed > no father collar worker > labourer or unemployed, no father collar worker > labourer or unemployed > no father FATHER S EMPLOYMENT CONTRACT permanent, self-employed> fixed-term, permanent, fixed-term, self-employed permanent > self-employed > fixedterm, none not significant none > none FATHER S STUDY QUALIFICATION degree or diploma > primary or lower degree or diploma > primary or lower degree or diploma > primary or lower degree or diploma > primary or lower secondary school, no father secondary school > no father secondary school, no father secondary school > no father SUPPORTS FOR STUDYING GEOGRAPHIC AREA Contextual variables MOTHER S PROFESSION MOTHER S EMPLOYMENT CONTRACT MOTHER S STUDY QUALIFICATION TABLE 6.1: Contextual variables and TECO results per Geographic Area and managerial/professional or whitecollar worker > labourer or unemployed, no mother not significant degree or diploma > primary or lower secondary school, no mother managerial/professional or whitecollar worker > labourer or unemployed, no mother Correlations with TECO permanent > fixed-term, self-employed > none degree or diploma > primary or lower secondary school, no mother managerial/professional or whitecollar worker > labourer or unemployed > no mother permanent > fixed-term, selfemployed, none degree or diploma > primary or lower secondary school > no mother managerial/professional or whitecollar worker > labourer or unemployed > no mother permanent > fixed-term, self-employed > none degree or diploma > primary or lower secondary school > no mother SCHOLARSHIP not significant non beneficiary > beneficiary not significant non beneficiary > beneficiary STUDENT RESIDENCE nd not significant nd not significant MEAL VOUCHERS not significant nd not significant non beneficiary > beneficiary STUDENT COLLABORATION CONTRACTS not significant under contract > no contract nd under contract > no contract COURSE WITH ADMISSION TEST National admission test or local admission test for 1% of students > no admission test, local admission test National admission test or local admission test for 1% of students > no admission test > local admission test National admission test or local admission test for 1% of students > no admission test, local admission test National admission test or local admission test for 1% of students > no admission test > local admission test but for less than 1% of students INDIVIDUAL MERIT EXTERNAL MERIT FOREIGN LANGUAGES KNOWN at least one > none at least one > none at least one > none at least one > none NUMBER OF UNIV. COURSES FOLLOWED IN FOREIGN LANGUAGE, IN ITALY none > at least one none > at least one none > at least one none > at least one NUMBER OF UNIV. COURSES FOLLOWED IN FOREIGN LANGUAGE, ABROAD not significant not significant at least one > none not significant NUMBER OF MONTHS ERASMUS OR OTHER PROGRAMME not significant at least one > none at least one > none at least one > none HIGH SCHOOL DIPLOMA GRADE (+) medium (+) medium (+) medium-low (+) medium HIGH SCHOOL TYPE classical or scientific lyceum, other institute > other lyceum > technical or vocational institute classical or scientific lyceum > other institute > other lyceum, technical or vocational institute classical lyceum > scientific lyceum, technical or vocational institute, other lyceum, other institute classical or scientific lyceum > other institute > other lyceum, technical or vocational institute MEAN GRADE IN UNIVERSITY EXAMS SAT (+) medium-low SO FAR (+) hardly significant (+) medium-low (+) hardly significant QUALITY OF EDUCATIONAL ENVIRONMENT - VQR R12 (+) medium (+) medium (+) medium-low (+) medium SELF-ASSESSMENT OF EDUCATIONAL ENVIRONMENT - EXPECTED COMPETENCES ( SUA FORM) QUALITY OF STUDENT ENVIRONMENT - M (+) medium INDEX (+) medium (+) medium-low (+) medium-low SELF-ASSESSMENT OF STUDENT ENVIRONMENT - ATTENDANCE REPORTED not significant not significant not significant not significant AS REGULAR SELF-ASSESSMENT OF STUDENT ENVIRONMENT - COMPETENCES not significant ACQUIRED AT UNIVERSITY PERCEIVED AS not significant not significant RELEVANT FOR THE TEST BY STUDENT <: TECO result significantly different versus (95% confidence interval) if sample size > 3. not significant: no significant difference between TECO scores (95% confidence interval) if sample size > 3. nd: Number of students who answered or who are included in the condition of the variable being higher or equal to 3. competences perceived as not relevant > relevant, but difference not significant (+): Positive simple correlation with TECO calculated on the means per Disciplinary Group, except for the variables AGE and TIME SINCE DIPLOMA for which it is calculated on raw data. Correlation levels are categorized as follows (in ascending order): barely significant (.2); medium-low (.21.4); medium (.41-.6); medium-high (.61-.8); high (.81-1.). (-): Negative simple correlation with TECO calculated on the means per Disciplinary Group, except for the variables AGE and TIME SINCE DIPLOMA for which it is calculated on the raw data. Correlation levels are categorized as follows (in ascending order): barely significant (.2); medium-low (.21.4); medium (.41-.6); medium-high (.61-.8); high (.81-1.). >: TECO result significantly different versus (95% confidence interval) if sample size > 3 (**): The columns NORTH, CENTRE and SOUTH show only those Disciplinary Groups for which more than 3 students participated in the pilot test. All Groups that are not shown had less than 3 students, except the Artistic Group, which was not present in the SOUTH. Source: See TABLE 3 Variables for which the source is the student are shown in grey background. 7

71 71

72 4.8 The influence of the family s socio-cultural condition The influence of parents appears in the sense that an absent mother (not father) lowers the TECO, all other things being equal, and having a father employed in a managerial/professional position (but not a mother) raises it. The effect of the socio-cultural condition is much stronger in simple correlations, because in multiple regressions it is also exercised through diploma and University grades, as well as in the choice of secondary school. It can be seen, therefore, that some contextual variables such as, for example, family status lose value once others are controlled. This is specifically because family status helps to predict the type of secondary school diploma, the diploma grade, the type of course of study chosen and the average University grade, in addition to predicting the result on the TECO test. As shown in Table 7.1, the sign of the coefficients on these variables is as expected, but their statistical significance disappears both individually (except for the cases cited above), and jointly (the F tests for the inclusion of the group of variables related to parents study qualification or profession do not refute the null hypothesis of their absence of additional explanatory capacity). On the other hand, in simple correlations, a high professional and cultural status of the parents (see Table ) strongly correlates with success in the TECO: when the mother has a managerial/professional position or a white-collar job, has a University degree or high school diploma, regardless of the father's position, results above the mean and the median are observed; and this applies equally to the father. The absence of at least one parent is obviously a deprivation condition, and the worst one, much worse than having a father or mother who is a manual labourer, unemployed or unqualified. 72

73 4.9 Other social and family information The decisive superiority of simple correlations with respect to multiple regressions is seen only where the latter are missing for some of the contextual variables, perhaps because these regressions are too parsimonious or because for some of the data the number of observations is considerably lower (as shown in Table 3), so that more sophisticated analysis is not recommended. The examination of simple correlations between all contextual variables and the result on the TECO or on its two components, complemented at times by looking at indirect correlation (e.g. with diploma and University grades), yields some broad generalizations, not necessarily applicable to all the geographical macro-areas of the country (see Table 6.1 already shown). Looking at the variables for family data, it is somehow surprising that the cases where there are siblings at the University or not are observationally equivalent (see Table 6.1.2) and likewise for living off-site with respect to the University or not (see Table 6.1.5). The size of the family seems instead to have a negative effect (see Table 6.1.1), and likewise for the travel time required to reach University (see Table 6.1.8). 73

74 74

75 75

76 Students with more technological equipment on average perform better (see Table 6.1.9), as well as those who go on at least one trip per year outside the region (see Table 6.1.1) or abroad (see Table ); this does not seem to influence the mean diploma grade, but it does influence the mean grade on University exams sat so far. 76

77 4.1 Supports for studying and individual merit Not surprisingly, the TECO score drops if the student also works (see Table ) and the various types of support for studying do not compensate for the disadvantages of different kinds affecting those students who usually have recourse to support (see Tables ). The only type of support that helps raise the TECO result seems to be the student collaboration contract (see Table ). This is the only one assigned strictly on merit-based criteria (without consideration for the condition of poverty) but unfortunately it also concerns few students. 77

78 All the meritocratic-type contextual variables tend, in general, to be significant on the TECO result and scores. There is instead no significant difference between tested students who passed a University admission test versus those who did not (see Table and following). However, this only depends on the fact that this distinction as such is not fully meaningful For example, among the best in the absolute sense in the TECO are Medicine (with a National Admission test) and Psychology students (among whom those who sat an admission test at the time of entry represent 1% of the population). This prevents a comparison with others who accessed the same courses without an admission test. Vice versa, in the case of Mathematics-Physics-Statistics, among those tested only 26 sat an admission test, a number so insignificant as to make the comparison with graduating students who did not sit an admission test of little interest. Ultimately, the results in the TECO seem to be, on average, better for those who, among the graduating students, passed a national admission test or a local one for 1% of entering students. In second place, the TECO performance seems higher for those, among third and fourth year students, who belong to disciplines with no admission test (e.g. History, Philosophy and Law). Those who came last are the students enrolled in hybrid disciplines, with or without a local admission test, where there are normal cases, such as Chemistry (for which the graduating students who had to pass an admission 78

79 test were better than those who did not) as well as paradoxical cases, such as Engineering (for which the graduating students who had to pass an admission test performed on average worse than those who did not) Students self-assessment of the competences they have acquired Of particular interest is the examination of the connection or lack thereof between the level of generic competences acquired during University studies (as perceived by the tested graduating students) and the level of performance on TECO. In fact, there is no positive correlation between students perception that they have acquired competences and their results (see Table 9.5 and following). The best students are the Socratic ones, who know that they do not know more than the average Italian and in reality obtained the highest scores (in the North-West quadrant of the graph on the left), for example the eligible students in Medicine and Psychology, while the more arrogant are those who think they know and are right, such as students of Mathematics-Physics-Statistics (in the North-East quadrant). The most incoherently satisfied with their course of study are the pre-socratic students (e.g. those in Sociology) who do not know that they do not know (in the South-East quadrant): they claim to have the competences whereas in reality they perform poorly. Conversely there are modest students who coherently know that they do not know and in reality do not know, such as those in Education. It would thus seem legitimate to conclude that students perception that they have acquired competences (quite high in Italy, expressed by a sizeable 8.45% of the tested graduating students) is indicative only of high customer satisfaction (particularly in the southern Universities) but of nothing else of objective character (see Table 9.5). 79

80 8

81 It only rarely happens, and only for those in the highest quartiles at (see Table 9.9), that there is coherence with a low statistical significance between self-assessment of competences and measurement of these competences via TECO. 81

82 4.12 Initial estimates and corrections for contextual diversities It is relevant at this stage, on unpolished data such as these, to consider two different questions. Firstly, whether the better performing Universities would maintain their lead after correcting the results on the TECO for various differences in the characteristics of students as from their entry to the University. Secondly, and more importantly, what would happen if the current (positive) self-selection bias were corrected for example, the participation rate of Bologna, the smallest one registered, is 5 percentage points away from that of Udine. What would the results have been, in terms of assessing the generic competences of graduating students, if only 13.9% of those eligible in Udine had turned up or if the test had been administered to 64.1% of those eligible in Bologna? The answer to the first question lies in the econometric analysis by Peracchi (214) (see Table 7.1 mentioned above). Once the condition of all other things being equal is met via multiple regression (except for the self-selection bias evidenced by substantial heterogeneity in the P index), the fixed effect is maximum for the University of Udine and solidly positive for the Universities of Bologna, Milan, Padua, Florence, with the addition of Eastern Piedmont, which like Udine registered the highest participation in the test. The examination of the fixed University effects confirms a clear distinction, which is already evident in simple correlations, between on the one hand the Universities of Central- Northern Italy, excluding those in the Rome area (Bologna, Florence, Milan, Padua, Eastern Piedmont and Udine), and on the other hand the Universities of Central-Southern Italy (Naples, Lecce, Messina, 82

83 Cagliari, Rome La Sapienza and Rome Tor Vergata) 33. For the first group, the fixed effects are always positive and statistically significant, while for the second one they are negative and statistically significant in the case of Naples, Lecce and Messina Initial estimates and corrections for the self-selection bias Turning to the second question, to give an idea of what would happen for example in the comparison between Universities or between Disciplinary groups if the self-selection bias were removed, we first demonstrate that this bias is positive. Indeed, all variables positively correlated with the TECO result are higher among eligible students who sat the test versus those who did not and vice versa for the negatively correlated variables, which are lower among eligible students who sat the test. Therefore, if the self-selection bias were corrected the comparative advantage would increase for those realities where the participation index, P, in the test is higher. For example, the analysis depicted in Table 8.1 indisputably shows that the diploma and University grades of students who sat the test are significantly better than those of eligible students who did not sit the test and those of ineligible students. For this aspect, therefore, the self-selection bias is presumably positive and this is so in almost all Disciplinary groups (see Table 8.2) and almost all Universities (see Tables ). 33 Moving up through the TECO percentiles, the relative advantage (positive fixed effect) of the Universities of Bologna, Eastern Piedmont and Udine, and the relative disadvantage (negative fixed effect) of the Universities of Naples and Tor Vergata diminishes, while the relative disadvantage of the University of Messina increases. Finally, as regards the comparison between the two Universities in the Rome area, the relative disadvantage of Tor Vergata decreases and ceases to be statistically significant when moving up through the percentiles. 83

84 84

85 This distortion occurs equally in all or almost all distribution quartiles (see Table. 8.7). The question hence arises of how much the results of some Universities with a low P index would go down if the participation rate were to increase, with the inclusion of pre-registered students or eligible students who did not show up for the test. For example, looking at Table 7.1, all other things being equal, the TECO score seems to decrease by points for every reduction of 1 point in the mean grade in University exams. Therefore, if the decrease was.17 points (as indicated by Table 8.1 in the comparison of means between eligible students who came to sit the test and those who did not), all things being equal, the TECO score would thereby change from the current mean level of to the potential level of ( = -2.7) The distortion in the TECO results, implicit in the self-selection of students who sat the test with respect to all other eligible students, will certainly disappear once, as ANVUR recommends, the generic competences test becomes standard practice for all graduating students and all Universities. Passing the test will not be a requirement for periodic accreditation, but sitting it before the degree will be required. Of course, in the future other changes of great significance for assessing the path of studies in University may be introduced, affecting students that have been enrolled for three or four years. Such changes could include combining disciplinary tests with generic tests, assessing competences upon both entering the University and exiting from it, and overcoming the opposition from the Guarantor of Privacy through appropriate regulatory or legal changes, so that Universities may receive the individual results of their students in nonanonymous form. 85

86 More generally, the self-selection bias is confirmed to be positive for all other contextual variables (age, citizenship, non-italian language, working student, female gender, residence off-site, etc.) that are significantly correlated, positively or negatively, with the TECO result. The eligible students who came to sit the test are on average systematically better equipped than those who did not, i.e. they show a higher presence of success factors and/or a lower presence of failure factors (see Tables 8.1 and 8.16). 86

87 87

88 88

89 4.14 Externalities of merit As partially already shown above, simple correlation analyses are useful for examining three issues: 1) to check the influence of some contextual variables of sociological nature, previously indicated with reference to social data, which econometricians are usually not very sensitive to; 2) to offer a few preliminary elaborations on the sign and amount of the correction for the self-selection bias, which makes it difficult to compare results between Disciplinary groups or Universities as well as between different countries 35 ; 3) finally, to examine some of the factors concerning the influence on the TECO results of externalities due to the student and academic environment. On this latter aspect, it appears that the quality of the student environment, approximated by the merit index M (which increases with the mean diploma grade, VMD, and decreases with the mean grade in University exams, VME), is correlated, albeit weakly, with the TECO result in the various Disciplinary groups (see Table 6.3.6). However, this correlation becomes negative with respect to the Universities (see Table 6.3.1) because of the inflation grading existing in both diploma and University grades particularly in the South (it should be noted that Cagliari is an exception; it should also be noted that Rome 1 is conversely very rigorous in this respect). 35 Not forgetting that the sample which sat the CLA+ in the United States is not larger than the sample which sat the TECO in Italy, but the college students who sat the CLA+ are not self-selected: they are selected at random or not selected at all. 89

90 9

91 The positive externality on the TECO created by a high-profile academic environment seems to be instead much stronger. There is an excellent correlation (.6-.8) between the TECO results and the scientific value of the teachers involved in teaching courses or in the Universities of the pilot phase, as indicated by R12, derived from VQR (see Tables and 6.3.3): ex post, the quality of teaching outcomes has a very good match with the quality of the research results of University teachers. 91

92 An equally good match also exists between the teaching outcomes shown by TECO and the engagement shown by teachers in a form called SUA as regards setting objectives in terms of expected results in generic competences. This is demonstrated in Tables 9.2 and 9.3. First of all, it emerges that the more frequent evaluations are B for TECO and A for the SUA form, so that the combination (B;A) occurs in 42% of the cases examined. On the whole, it should be pointed out that in one third of the cases 36 there is total concordance between the qualitative ratings of the results achieved ex post in the TECO in different classes in different Universities, and of the ex ante formulations of expected outcomes in the SUA form. Also, the cases of strong discrepancy between these two forms of evaluation are rare, amounting to less than 8.5%. 36 The cases examined do not include those in which one of the values is "null" or the SUA rating is not univocally defined (for example, cases of a B/C rating). 92