EVALUATION STUDY OF A SET OF CHILEAN DOCTORAL PROGRAMS November 24, 2014

Transcription

1 EVALUATION STUDY OF A SET OF CHILEAN DOCTORAL PROGRAMS November 24, 2014 (Credits and Acknowledgements) PRESENTATION This Evaluation Study is part of a set of efforts undertaken by the Higher Education Division and the Institutional Financing Department of the Chilean Ministry of Education, in order to enhance and strengthen national doctoral programs, namely, a set of 93 Chilean doctoral programs selected for compliance with specific requirements in terms of consolidation. The purpose of the Study is to contribute toward greater knowledge of almost one hundred of the most established doctoral programs in the country: (a) for ongoing management and improvement of programs by the institutions; (b) for the design or enhancement of public policy and other actions aimed at better developing programs both nationally and internationally; and (c) to facilitate comparisons in relation to parameters that may be of interest among programs of differing conditions and stages of development. This is the first time there will be information available in Chile that has been gathered through the very universities that house this ample collection of programs. This was accomplished by consulting their highest authorities, the director of each program, each faculty member, as well as each student. Also part of this report are a set of studies carried out by the National Commission for Scientific and Technological Research (CONICYT) titled Scientific Output and Impact of Academic Participants, FONDECYT Research Project Financing to Academic Participants and CONICYT Ph.D. Scholarships assigned to student participants. Additionally, a study titled Scientific Output of Ph.D. Students Graduated from the Participating Ph.D. Programs in the Period is included. In terms of methodology, it was deemed advisable to consider an international benchmark that uses with appropriate adaptations the methodology employed by the National Academy of Sciences (NAS) and the National Research Council (NRC) of the USA in A Data-Based Assessment of Research-Doctorate Programs in the United States (2010). To this end, a pilot experience was carried out in Chile in 2012, to a deliberate sample of 14 programs belonging to nine universities ( Final Report of Pilot Program: Analysis of Chilean Ph.D. programs using the U.S.A. Ph.D. Assessment Methodology ). With the lessons learned from this experience, the process was scaled-up and now concludes in this study with the analysis of 93 programs belonging to 15 universities. 1

2 From an institutional viewpoint, this study was advised by a Technical Committee comprised of professors Jorge E. Allende (Universidad de Chile), Andrés Bernasconi (Pontificia Universidad Católica de Chile), and Luis Morán (Universidad de Concepción), selected for being faculty in various areas of knowledge and for belonging to the three universities with the highest number of selected programs, and by doctors Ricardo Reich and Diana Veneros, of the Higher Education Division. Dr. Leonardo Epstein acted as statistical consultant. The entire team has been constantly advised and in direct contact with the highest authorities of the National Academy of Sciences. EXECUTIVE SUMMARY (In process) I. INTRODUCTION (In process) II. BACKGROUND The two most important references for this Study are the NAS/NRC assessment of doctoral programs in the USA, and the pilot project carried out in Chile in The assessment by the National Academy of Sciences (NAS) and the National Research Council (NRC) of the USA was administered to almost 5,000 research doctorates at 212 institutions of higher education in that country. The final Report is titled A Data-Based Assessment of Research-Doctorate Programs in the United States (2010) 1. Information about the pilot project experience carried out in Chile (August-December 2012) and the results were summarized in Informe Final. Proyecto Piloto. Análisis de programas de doctorado chilenos usando la metodología de la National Academy (NAS) y del National Research Council (NRC) de los Estados Unidos de Norteamérica 2. This Final Report states that the pilot experience achieved its main objectives and that the project was received very positively and there was consensus with regard to its relevance and usefulness (p.13). As to the trial under local conditions, the experience showed the need to 1 National Research Council of the National Academies. Committee on an assessment of Research- Doctorate Programs in the United States, J. P. Ostriker, C. V. Kuh, and J. A. Voytuk, Editors. The National Academies Press, Washington, D.C. Available at 2 División de Educación Superior, Ministerio de Educación de Chile, diciembre Dr. Jorge Allende, Dr. Rafael Benguria, Dr. Andrés Bernasconi, Dr. Luis Morán. Available at 2

3 improve instruments for gathering information (questionnaires) and data analysis, and improve communications with the players involved in the process, which was done for the final part of this scale-up. III. METHODOLOGY As indicated in the Introduction, this NAS/NRC Evaluation Study of National Doctorates was based on the methodology used for the assessment of the USA national doctorates. The process in the USA lasted four years, and hence the publication is dated The main elements of the NAS/NRC methodology adapted for this study are: - Considering each doctorate program as a unit of analysis, and not the academic department or unit to which it belongs. This is on account of agreement with the idea that despite the difficulties in classifying and assigning (faculty to each program) the program continues representing a unit that more exactly defines the sphere of doctorate student experience once admitted to a particular department or program 3. - Working with faculty organized into three categories: core faculty, associate faculty, and new faculty. Albeit this classification is not current practice in the Chilean system, its use was indispensable in order to maintain comparability with the study in the USA in relation to faculty academic productivity. - Gathering information through on-line document review and consulting four types of players: institutions, programs, faculty, and students. - Employing an array of instruments for information gathering, consisting of questionnaires based on those used in the USA (for comparability purposes), and which were adapted to the national context, nevertheless procuring to keep the association albeit approximate with the respective questions used in the USA questionnaires. The Study, in its full scale stage, was designed as a process to be held in phases, allowing simultaneous implementation in certain cases. The phases were: (A) Designing initial work; (B) Improving the adaptation of questionnaires used in the NAS/NRC assessment, and designing the use of questionnaires and statistical processing of the information gathered; (C) Designing, jointly with CONICYT, a study on Scientific Production and Impact of Faculty participating in the Evaluation Study and studies about FONDECYT project activity of faculty and Scholarships for Doctorate students; (D) Designing at the Higher Education Division a study on Scientific Production of Ph.D. candidates that graduated from the participating program during ; and (E) Designing communications with the players involved; 3 National Research Council of the National Academies (op. cit.) 3

4 Phase A: Designing initial work This phase considered the following main actions: (a) Analyzing existing information; (b) Setting eligibility criteria for the programs in the study; (c) Designing work with program directors for the organization of core faculty according to the NAS/NRC methodology; (d) Defining the structure and content of the Results Final Report. a. Analyzing existing information The two main sources analyzed for the Study were the assessment experience of USA doctorate programs and the pilot project experience in Chile (August-December 2012). In the case of the pilot experience, the analysis included minutes of meetings, workshop materials, and e-files with mailings and other texts pertaining to the pilot experience, as well as informative materials of the doctorate programs of various universities. Likewise, meetings for analysis were held with faculty members of the Technical Committee for the experience, doctors Jorge Allende, Andrés Bernasconi, and Luis Morán, and the leaders of the process at the Higher Education Division, Dr. Ricardo Reich and Dr. Diana Veneros, and Mr. Jesús Marolla, who acted as Assistant. b. Setting eligibility criteria for the programs in the study As determined in the pilot project, it was decided this first evaluation study of national doctorates would be administered to a set of programs selected in accordance to eligibility criteria related to a degree of consolidation in terms of their development. The eligibility criteria for the doctorate programs were the following: a. Accreditation for at least four years. b. At least five graduates in the last five years (2008 to 2012). c. At least 10 students enrolled in the program at April In view of the size of the student population in certain programs, 3-year accredited programs were also included, provided they had: - More than ten graduates in the last five years (2008 to 2012), and - More than fifteen students enrolled at April It should be noted here that applying these criteria resulted in a final list of 93 programs belonging to a total of 15 universities. Five of these are joint programs between two institutions 4

5 and program direction is alternated between both institutions. Hence, for the purposes of this Study, the program representative was the program director at the university directing the joint program at that moment. UNIVERSITIES and PROGRAMS (Box with a list of Universities and programs) c. Designing work with program directors for the organization of core faculty according to the NAS/NRC methodology For comparability of certain data obtained in this study with those analyzed by the USA NAS/NRC assessment, it was necessary to use the rules defined for that assessment to define doctorate program core faculty. These rules distinguish three types of faculty within doctorate programs: core faculty, associate faculty, and new faculty. This distinction is based on the following principle with which everyone responsible for the USA NAS/NRC assessment agreed: there should be no double counting (of faculty) and avoid universities assigning their most prolific and distinguished faculty to multiple programs 4. This principle also seemed reasonable for the case of Chile, and was therefore adapted to the reality in the country. Hence program directors were asked to in conversation with faculty put together the lists of core faculty according to the established requirements, the most important of these being the faculty s commitment to being considered core faculty only in the program to which they dedicate 50% of their time. In line with the above, the requirements considered for each category are presented below. Core faculty are program faculty that jointly fulfill the following requirements: 1. Have mentored or are mentoring at least one doctoral thesis in the program or have in the last five years ( ) being part of at least two Doctoral Examination Boards; 2. Have at least 22 hours p/week at the institution that hosts the program; and 3. Are not part of core faculty (as defined) of another doctorate program. Associate faculty are program faculty that jointly fulfill the following requirements: 1. Belong to core faculty of another program, but have mentored or are currently mentoring at least one program thesis, or have been part of degree examination boards during the last five 4 National Research Council of the National Academies (op. cit.) 5

6 years ( ); and 2. Have at least 22 hours p/week at the institution that hosts the program. New faculty are program faculty that jointly fulfill the following requirements: 1. Belong to the program; 2. Have at least five years seniority with the doctoral degree; 3. Have at least 22 hours p/week at the institution that hosts the program. Note: Faculty that belong to more than one doctoral program should specify the percentage of their scientific production (publications, citations, and distinctions) that they wish to assign to each program, considering they should assign at least 50% of this productivity to the program to which core faculty they belong. They could assign the remaining productivity to the program or programs in which they are associate faculty. d. Defining the structure and content of the Results Final Report Together with the work plan, and for guidance, a provisional structure was provided for the Results Final Report, identifying the principal content. Phase B: Improving the adaptation of questionnaires used in the NAS/NRC assessment, and designing the use of questionnaires and the statistical processing of the information gathered The main strategy for this study consisted in gathering information using four questionnaires: One questionnaire for the institution, to be answered by the authority responsible for the doctorates (research vice rector, academic vice rector, or the graduate authority responsible for doctorates at institutional level). A questionnaire for the doctorate program (to be answered by the director or coordinator). A questionnaire for each faculty member in the program s core faculty, in any of the categories recognized by this Study, and A questionnaire for each student enrolled in the program at the time of administering the questionnaire. After analyzing population sizes and other statistical considerations, it was decided not to work with samples and instead administer the questionnaires to the representatives of all the universities, all program directors, all faculty, and all students. 6

7 Given that after administering the pilot project for the evaluation study a certain degree of dissatisfaction was detected with the formulation and wording of some questions, it was decided to improve the questionnaires used in the pilot, designing and administering a text search process and which resulted in the administering of the questionnaires to deliberate samples of the subjects belonging to the four target groups. With regard to the design of questionnaire implementation and the statistical processing of the information gathered, it was decided to: (a) work with complete populations; (b) gather the information of the individuals/members of each population surveyed between the months of August and October 2013, considering the following criteria: full name, ID number (when applicable), and ; and (c) conduct pilot tests with members of the advisory committee and subsequently groups of individuals/ of the population. Phase C: Designing, jointly with CONICYT, a study on Scientific Production and Impact of faculty participating in the Evaluation Study and studies about FONDECYT activity of Faculty and Scholarships for Doctoral students It was agreed that CONICYT would carry out a study on the scientific production of core faculty of the programs being evaluated. This is a comprehensive study of scientific production and impact of the publications of the faculty taking part in the Evaluation Study, in that it records for each respondent researcher the number of publications with their respective citations indexed in Web of Knowledge -Thomson Reuters, as well as publications in the five collections presented by WOS and SCOPUS. In addition, CONICYT implemented a virtual platform with information on the scientific productivity of faculty by institution, program, type of core faculty, and name of researchers. This platform will be made available to the national academic world after publishing this Report. For the Study, the name of each researcher was checked, recording the full name, origin, area of research, and current institutional location. It was agreed to only record those papers that could be confirmed as authored by the searched researcher, avoiding publications that could lead to error. The period considered for the references is 2003 to The information is presented on one sheet per author with their publications, in the form of evidence and proof of the search. 7

8 CONICYT Study on Scientific Production and Impact Confirmation of author and publication matches The result of the study is a productivity report for 1946 researchers searched using the Web of Knowledge database. The researchers belong to Core Faculty of national universities and the database search period is 2003 to 2012 inclusive. The publications considered were from the five collections presented in WOS (Science Citation Index; Social Sciences Citation Index; Arts & Humanities Citation Index; Conference Proceedings Citation Index - Science; Conference Proceedings Citation Index- Social Science & Humanities), in addition to other indexed document types. Researchers identified with publications in Web of Knowledge: 1,762 faculty appear recorded with publications. Researchers with no recorded publications in Web of Knowledge are 184, of which 55 record publications in Scopus. Search method As many as four searches were made using full names and surnames, name initial, and address of the author. Verification of scientific area, university, school and department recorded in the publication entry In case of researchers with the same surname and name initials, same institution and same scientific area, the group of regular co-authors was checked to assign the publication. Special authentication steps were taken for authors with traditional surnames. Search sources: Web of Knowledge and 8

9 Phase D: Designing at the Higher Education Division a study on Scientific Production of Ph.D. candidates that graduated from the participating program during In this study, scientific output (with full references) on Ph.D. candidates graduated from the participating programs in the period , as indexed by World of Knowledge, was requested directly to the program directors. Complete information was obtained for the 93 participating programs. Phase E: Designing communications with the players involved In this Study, special importance was assigned right from the start to communication among the players involved, including: University authorities, program directors, faculty, and students Authorities of the Higher Education Division of the Chilean Ministry of Education Authorities of the USA National Academy of Sciences (NAS) that gave their support for the pilot project initiative and on this occasion also accepted to support the scaleup of the experience with 93 selected programs. These communications were scheduled using various media such as official letters, s, sending of documents, and phone calls. With regard to face-to-face meetings, these were mainly scheduled with the institutional representatives and program directors, and set considering at least: one initial meeting and another for the final presentation of results, also attended by representatives of the USA National Academy of Sciences (NAS). By means of official letters dated 2013, a new relationship between the Higher Education Division and NAS authorities was formalized for accompanying the scale-up of the pilot experience carried out in 2012, sending the latter ad hoc information with regard to the design of the scale-up experience and its main features. IV. IMPLEMENTATION 1. INITIAL WORK Analysis of the NAS/NRC assessment and of the pilot project experience carried out in Chile with 14 national doctorate programs provided very valuable information for the design and implementation of the scale-up experience, seeking to evaluate the 93 programs surveyed by the Study, and the results of which are presented in this Report. 9

10 Program selection After applying the established eligibility criteria, 88 institutional programs and 5 joint programs between two institutions were selected. The majority of programs selected (60%) were concentrated in three universities: Universidad de Chile (in Santiago), Pontificia Universidad Católica de Chile (in Santiago), and Universidad de Concepción (in Concepción). Of these programs, 87% are hosted by universities of the Council of Rectors of Chilean Universities (CRUCH), and 13% by universities outside CRUCH; and, in terms of the type of institutions represented, in this study there are 7 State universities (47%) and 8 private universities (53%) From the viewpoint of doctorate program accreditations, these 93 programs represent 62% of currently accredited programs (149 programs at 12/08/2014). In terms of specialty types, the selected programs belong to seven of the eight broad fields of the Organization for Economic Co-operation and Development (OECD), as proposed in Fields of Education and Training 5, with a 52% concentration in field 4. Table x. Number of programs by broad field according to the OECD classification (Fields of Education and Training) Fields No. of Relative Values programs 1. Education 1 1% 2. Humanities and Arts 12 13% 3. Social Sciences, Business and Law 6 6% 4. Science, Mathematics and Computing 48 52% 5. Engineering, Manufacturing and Construction 13 14% 6. Agriculture and Veterinary 8 9% 7. Health and Welfare 5 5% 8. Services 0 0% % Taking Field 4 (Science, Mathematics and Computing) as total collective, which encompasses 48 of the 93 programs (52%), Narrow Fields also show a concentration in Biology and Biochemistry, as well as in Physics, as can be seen in the details provided in the following Table. 5 Fields of Education and Training Manual. Ronnie Andersson, Anna-Karin Olsson, Sweden Statistics, (ISCED, in close cooperation with UNESCO and OECD) 10

11 Table x. Field 4: Science, Mathematics and Computing Programs by Narrow Fields Field 4 programs by Narrow Fields 42. Life Sciences 421 Biology and Biochemistry 422 Environmental Sciences No. of programs 16 6 Relative Values 33% 13% 44. Physical Sciences 441 Physics 442 Chemistry (except Biochemistry) 443 Earth Sciences 46. Mathematics and Statistics 461 Mathematics % 10% 6% 13% 2% 462 Statistics 48. Computing 481 Computer Science 2 4% % Organization of core faculty Organizing core faculty into three categories unlike the traditional hierarchies was first explained in writing and subsequently discussed with program directors, eventually reaching agreement within the working groups at the initial workshop of the process held on 10/07/2013. Some of the difficulties that arose were related to the complexity of certain programs especially in the case of interdisciplinary programs or resistance by some faculty to necessarily belong to the core faculty of a single program, or to the calculation of 50% of their production. Other difficulties stemmed from a possible lack of timely communication between program authorities and faculty, which in some cases lead to unilaterally decided allocations of faculty to various categories. Lastly, there could even have been a lack of understanding as to how the requirements were to be applied. These difficulties lead to many exchanges by , and the core faculty organization process was finally closed by sending the final lists to all program directors asking for their review and possible amendments prior to starting work with the questionnaires. Despite this, implementation of the questionnaires revealed several cases when it was necessary to make further adjustments to the lists. Nevertheless, there may still be several cases where the core 11

12 faculty has not appropriately organized. The most usual error has been the presence of one same faculty member in more than one core faculty. 2. IMPROVING THE QUESTIONNAIRES Improving the questionnaires adapted from the NAS/NRC Assessment and used in the initial pilot required very careful work and a significant investment of time. A comparative table between the pilot project version (Chilean version) and the original NAS/NRC (USA version) was constructed for each questionnaire. Based on this, the Advisory Committee added, deleted, or improved the questions in the Chilean version, achieving a Chilean version 1. This was submitted to program directors and university graduate course directors at the workshop held on 10 July The groups formed were subsequently consulted by means of an active virtual dialogue. This resulted in changes to several questions, improvements to the questionnaire text, and the addition of several questions from the original NAS questionnaires which had not been considered in the case of the pilot experience. This process resulted in a Chilean version 2, which was once again reviewed by the Advisory Committee, giving rise to the Chilean version 3, tested with a deliberate sample of subjects from each of the four questionnaire populations. 3. STUDIES CARRIED OUT BY CONICYT 4. ON-LINE ADMINISTERING OF QUESTIONNAIRES AND STATISTICAL ANALYSIS OF THE INFORMATION GATHERED The administering of pilots aimed to identify problems with the interpretation of the survey questions, logical inconsistencies, and difficulties with the software or logistical problems. Based on the suggestions and errors discovered with the pilots, the committee modified questionnaires and procedures. The questionnaires were sent to all individuals/members of each selected population, namely universities, doctorate programs, program core faculty, and doctorate students. As such, the study comprises four surveys. The questionnaires were sent out by with instructions for completion and a link for respondents to access the SurveyMonkey (version Gold; platform that lodged the questionnaires as well as the gathered information. The process began with the questionnaire for institutions, followed by students, then the questionnaire for faculty, and lastly the programs. The gathered information for all four populations was obtained between the months of September 2013 and March

13 Once the individuals/members answered the questionnaires, the data was lodged with SurveyMonkey and subsequently downloaded in spreadsheets, one for each survey. Analysis of each question resulted in the computed statistics. In attention to the type of response, the advisory committee determined the most appropriate analysis, as well as the bivariate analysis for each questionnaire. 5. COMMUNICATION WITH PLAYERS INVOLVED Communication with the players involved began with a formal invitation to the rectors of universities hosting the selected doctorate programs. In all cases the rectors accepted on behalf of the selected programs to take part in the study, and welcomed the initiative with interest. Once universities were committed to the study, letters and s were sent to the senior authorities specifically related to the doctorates; that is, Vice Rectors, Graduate Directors, and Directors of the doctorate programs. This correspondence was continuously maintained throughout the experience. In terms of face-to-face meetings, a workshop was held (Crowne Plaza Hotel, 10/07/2013) with institutional representatives and program directors. This workshop formally initiated the process, explaining the methodology in greater detail on the basis of examples obtained in the pilot phase and reflecting on the impact and scope of the study, as well as gathering perceptions and suggestions by the universities in relation to the proposed instruments. In addition to these meetings, other meetings of academic communities related to the doctorates were also used for this purpose, such as for example doctorate program meetings with the Higher Education Division (Divesup), in order to inform on the ongoing status of the process and answer questions. With regard to the relation with National Academy of Sciences (NAS) representatives, in addition to the correspondence maintained during the process with the NAS President, Dr. Ralph J. Cicerone, there were also two meetings in Washington DC: - Meeting between Dr. Jorge Allende and the NAS/NRC Advisory Group for the Chilean experience, (WDC, 27/04/ ) 6 This meeting was held on a Sunday, since this was the only available moment within the 151 st Annual Meeting de la NAS (WDC, April). 13

14 At this meeting, Dr. Allende presented the preliminary results found to date in a set of 14 institutional programs and two joint programs, each between two universities, six universities involved in total. The aim was to discuss the methodology used, the statistical processing, and the results obtained. The members of the Advisory Group Dr. John Brauman, Dr. Charlotte V. Kuh, and Dr. James A. Voytuk provided a positive assessment of what had been done so far and the results obtained. They in turn gave a series of recommendations which represented important guidance for the fulfillment of the last phases. The main recommendation was to make a general analysis of the information gathered from all faculty, and a more specific analysis of core faculty. - Meeting between Dr. Leonardo Epstein, statistical advisor to this Study, and Dr. James A. Voytuk, expert in statistics and who acted as Senior Program Officer for the NAS/NRC assessment (Washington DC, 22/05/2014). At this meeting they discussed the statistical processing so far, how to continue, and how to finalize the process and obtain the results. V. RESULTS 1. Institutions Consulting the institutions was carried out through a senior faculty authority at each of the 15 universities participating in the evaluation. Respondent authorities were five Vice Rectors (1 Research Vice Rector, 1 Research and Graduate Vice Rector, 1 Research and Advanced Studies Vice Rector, 1 Academic Vice Rector, and 1 Vice Rector of Economic and Administrative Affairs), as well as 10 Director-rank faculty, of which 7 hold the position of Graduate Director and variations (3 Graduate Directors, 1 Graduate School Director, 1 Undergraduate School Director, 1 Graduate General Director, 1 Graduate Studies Director), one holds the position of Academic Doctorate Director, one Graduate and Post Graduate Degrees Director, and one Graduate Teaching Coordinator. The questionnaire consisted of four parts, with a total of 19 questions: Part A Institutional health benefits; Part B Assistantship systems: teaching and research; Part C Institutionality; and Part E Support programs. A. Health benefits Almost all universities (80%, 12 of 15) report doctorate students have access to emergency health services at their university or these are hired by the university for doctorate program 14

15 students; however, this percentage drops to half (40%, 6 of 15) when reporting if the university has a health insurance. Of these six cases, one states it is insurance included in the fee, one states it is insurance offered through the university and that students contract voluntarily, and four who marked the Others alternative provided the following specifications: - State medical primary care insurance - Set of services similar to insurance, aimed at covering primary medical and dental care for undergraduate and graduate students, free of charge. It includes General Medicine and some specialties, in addition to Dental Care, Clinical Laboratory services, and health Promotion and Prevention. - Insurance paid by the University for students without grants to cover this expense. - Basic care insurance financed by the University. B: Assistantship systems: teaching and research A large number of the participating universities provide teaching assistantships and research assistantships to help financing of doctorate students: 87% (13 of 15) of universities have teaching assistantships, and 80% (12 of 15) have research assistantships. C: Institutionality This part considers the organizational support structure for doctorate programs and some aspects of program organization. A 93% (14 of 15) of the universities declare having a central administration unit responsible for policy setting and promotion of doctorate programs, reporting that in 11 cases (73%) this Unit has at least management ranking. In relation to time organization of each program s academic year, 80% (12 of 15) of the universities have their doctorate programs organized in semesters, 7% (1 of 15) in quarters, and 13% (2 of 15) in more than one modality. As can be appreciated, there is a broad range that fluctuates between 32 and 40 weeks, with one university in the 32-week extreme and two in the other 40-week semester extreme. As to the number of weeks in the school year of doctorate programs (including examination weeks), 7% (1 of 15) of the universities has a 32-week school year, 40% (6 of 15) a 34-week year, another 40% a 36-week year, and 13% (2 of 15) a 40-week year. As can be seen, there is a broad range that fluctuates between 32 and 40 weeks, with one university in the 32-week extreme and two in the other 40-week semester extreme. 15

16 In order to be considered a regular student, at 73% of the universities (11 of 15) doctorate program students should enroll once a year, whereas at 27% of universities (4 of 15) they should enroll each semester. Enrollment is required of all students in each academic period at 93% (14 of 15) of the universities, whereas one university (7%) reported the great majority of our students are released from enrolment. E: Support programs A 40% of universities states having institutional support units for doctorate students with special needs, which evidently represents relevant room for action in terms of equity. In terms of financing support programs with own funds, all universities declare carrying out activities with their own funds, albeit the only two activities carried out by 100% of universities with their own funds are the following: fee exemption scholarships, and support grants for attending congresses and other scientific meetings. The following Table provides details of other activities and the percentage of universities that carry them out. Table x. Support with own funds at each university Type of Support No. of universities Percentage - Fee exemption scholarships % - Support grants for attending congresses and other scientific meetings - Maintenance grants 14 93% - Financing for visiting professors 11 73% - Support grants for doctoral theses 10 67% - Scholarships for students stay abroad - Financing of minor capital investment for 8 53% students (e.g. offices, computers, desks) - Financing for thesis co-mentored programs 6 40% - Financing for the organization of symposiums or other scientific meetings - Scholarships for faculty stays abroad 5 33% - Grants for the purchase of textbooks and 3 20% software licenses Others: - Student stays abroad - Support for interdisciplinary theses % 7% 16

17 A 47% of universities (7 of 15) declare having equal opportunity policies for students. When asked about the specific minority groups where the equal opportunity policies are concentrated, 33% (5 of 15) answered none, 20% (3 of 15) answered ethnic groups, and 13% agreed on two groups: Persons with special needs, and Persons from special geographical areas. A 53% (8 of 15) marked the Others alternative, for which one university specified the group of foreigners and the other seven in various ways expressed everyone receives equal treatment and there is no discrimination. 2. Programs 3. Faculty Faculty were consulted by means of an on-line questionnaire administered to all faculty participating in the doctorate programs; that is, core faculty, associate faculty, and new faculty. The results of the responses given by core faculty are The number of respondent faculty varied according to the sections of the questionnaire Students VI. DIFFICULTIES, LIMITATIONS, AND LESSONS LEARNED MAIN DIFFICULTIES Most of the difficulties found in the design and development of this Study are proper to works of this nature, and in all cases it was possible to enjoy good disposition by the various players involved in order to resolve the issues. The most eloquent proof of this statement is that all programs persevered in the process and it was possible to complete the Study with the 93 initially committed programs. Among the difficulties, it is worth noting the following: a. An often insufficient level of development of program information systems. In several cases, the programs stated having to produce statistics and set-up records that would be useful in future. In others, there were several statements in the sense that the process would serve to put their house in order. b. Management issues in the case of directors, mainly due to an overload of duties, the little time available, lack of collaborating staff, and the challenge of directing a faculty team who 17

18 also have multiple duties and priorities that compete with the development needs of the doctorate program. c. In the case of faculty, the tension between teaching, management, and other duties. Same as for program directors, faculty also identified lack of time as the common trait resulting from work overload, the multiplicity of duties, and the lack of collaborating staff. Consequently, estimated time was insufficient and the deadlines had to be put back on account of very valid reasons, proper to the academic agenda of teaching and research. d. Difficulties associated to administering on-line questionnaires. Despite having used on this occasion better software than for the pilot experience, and also having taken into account the suggestions stemming from that experience, there were more inconveniences than expected and which were related, among other things, to the need to improve indications to users and employing more radical strategies for these indications to be truly read and followed. Furthermore, the filters used by universities for materials such as the software employed by the questionnaires proved to be very powerful and, in several cases, there was no timely detection of their existence and inhibitory action, whereby Divesup actually sent the questionnaires but they were not received by the recipients. e. Core faculty organization in the categories of core faculty, associate faculty, and new faculty presented difficulties in being accepted and, once accepted, in the implementation. Acceptance was given at the workshop held with all program directors and graduate directors of universities (10 July 2013) and subsequent information sent in writing. In addition, there were telephone communications and exchanges with program directors and graduate directors, as well as other authorities at various universities. Nevertheless, the process to obtain the lists of core faculty was slow, and in several cases there were allocations that made it necessary to check the final lists several times. f. One circumstance found repeatedly during the process was the lack of a formally established replacement when a director was absent. On several occasions this delayed tasks that could have been finished on time. Another difficulty was the evident lack of communication that existed in some cases, so much so that information took a very long time to reach faculty or the allocations to particular categories were made without consulting the faculty involved. 18

19 MAIN LIMITATIONS This Study has several limitations, including the following to be noted: a. The study shares a general and historical difficulty both in the country as well as in other parts of the world related to the prevalence of certain specialty fields, such as they have been called in the questionnaires for information gathering. The selected doctorate programs mostly belong to major Field 4 (Science, Mathematics and Computing) as classified by the OECD, which in many cases coincides with the programs that have achieved greatest development over time and that, in addition, correspond to disciplines for which there is greater development of the procedures and instruments for their management, measurement, and valuation. b. Given that base information for the Study was provided by the programs as such, the value of this basis is given by the information system of the respective universities. Despite the very good disposition encountered and the spirit of collaboration of universities and programs, the information systems show need for improvement that will probably be met more quickly thanks to the contribution implied by the Study in this respect. c. This is a Study that has managed to gather valuable quantitative information about the programs in terms of the existence or absence of particular attributes or conditions. Nevertheless, the degree of such attributes or conditions is not reflected. For example, there is information about whether particular faculty does or not mentor theses and their number in a particular period, but not about the attributes of this mentoring or the theses. d. In the case of the CONICYT study about faculty production and impact, the authors of the Study warned that a search was done for each faculty by the name that appears in the list sent by the program director; however, sometimes faculty publish under a different way of giving their name, adding or omitting a surname or other variables. Hence, some results may not be complete or some faculty may not even appear in the Study. Note: The website where this Report is published will be always open to possible remediation of this situation by means of notice given by the faculty as such in order to make the appropriate corrections or inclusions. 19

20 SOME LESSONS LEARNED A process such as the one followed by this Study is also a learning process. It provides everyone involved with an inside vision of each program, with information about its nature, history, development, and current situation, and also with the possibility for interaction at an academic level marked by a special disposition toward the search for further knowledge and understanding of what the program is and does, and why it is done the way it is done. It also provides grounds for reflection on possible improvements. The main lesson learned is an example of good disposition, seriousness, and the generosity encountered at the universities and programs. Although initially this Study appeared to many as yet another external initiative requiring the precious time they generally do not have, there was soon a good welcome, an open spirit toward providing information and, above all, demonstrating genuine interest in making the most of the opportunity to improve on the culture of self-evaluation forged through the accreditation processes. Shown below are the lessons learned, following the chronological development of the Study; that is, design, implementation, and results. 1. With regard to the Study design 1.1 It is clear that when designing the methodological aspects to be considered for the Study it is necessary in parallel to work on the implementation. This is a major issue, above all on account of the multiple duties and demands on program directors and faculty. In this sense, it would seem desirable to have more initial meetings and encourage early commitment. 1.2 With regard to time for the information gathering stage, it would appear as strategic to design the schedule for administering the questionnaires in line with the seasonality of academic work related to the calendar for teaching, research, presentation, implementation, monitoring, and closing of projects, as well as other duties proper to the faculty participating in doctorate programs With regard to the Study implementation and results

21 VII. OUTLOOKS The purpose of this Study has been to contribute toward greater knowledge for management and improvement of programs by institutions, for design and enhancement of public policy and other actions aimed at better developing programs both nationally and internationally. Also to facilitate comparisons in relation to parameters that may be of interest among programs of differing conditions and stages of development. In this context it is possible to state that results of two complementary natures were obtained, and which contribute in different ways to what is pursued: quantitative information about a diversity of aspects related to the programs, and qualitative type information that is principally referred to the opinions of players in relation to aspects they consider important in a doctorate program. 1. Main contributions by quantitative information: 1.1 With regard to university institutions: how many and which are the national universities that have the most established programs, how they are housed institutionally, information about the nature of the program and characteristics of the students, and the support offered in terms of attention to diversity and financing. 1.2 With regard to doctorate programs: the nature of the program in relation to specialty fields and narrow fields and the interdisciplinarity, composition of core faculty; and information about the students: selection, retention, graduation, and occupational destination, support offered to them, teaching activities of students and assistantships, strategies for recruiting foreign students, national or international agreements for comentoring theses, double degrees and other forms of joint work, and maximum capacity of the program for admitting students. 1.3 With regard to faculty: how many faculty comprise program core faculty and in what category, whether core faculty, associate faculty, or new faculty, and also according to the traditional hierarchies in Chile; specialty and sub-specialty fields of faculty; participation in program activities (teaching, research, committees, and others); prior experience and training trajectory, including post-doctorates; academic production; students under the responsibility of each faculty, at present and in the last ten years; and opinions about program strengths and opportunities for improvement. 1.4 With regard to students: certain elements for their characterization; type of support received, prior studies, trajectories followed and to be followed, as well as the time committed; participation in academic activities (congresses, internships, and others); publications prior to and during the doctorate study; and opinions about various aspects related to the quality of the program. 21

22 2. Qualitative information This mainly refers to the opinions of players in relation to aspects considered important to a doctorate program. In fact, this aspect is still very preliminary in this first experience; however, the responses and priorities indicated by faculty and students about aspects that contribute to the good quality of a doctorate program could be a valuable contribution to academic dialogue, which aside from resting on specialized knowledge about quality, may also provide insights on the thoughts of local communities in relation to these topics. The following is placed at the service of doctorate programs and interested researchers for future use: - database constructed with the information gathered - complete processing, the summary results of which is provided in this Study - responses obtained from faculty and students about the quality attributes of the doctorate programs studied - array of applied instruments and suggestions for improvement. If self-knowledge and understanding of what is a particular doctorate program is the basis for improvement and development, this Study may be of use inasmuch as it has focused on achieving first knowledge about a selected set of almost one hundred programs for which information is expected that should serve as baseline for future studies that aspire to understand the behavior of what is in existence, learn of its development over time, and to further elaborate on qualitative aspects that for the time being are recorded only in terms of presence or absence. Same as analyzing the results of regular studies of programs throughout their history often reveals a positive trend for example, in the development of disciplines, the number of students, number of graduates, time to graduation, increased diversity of the persons participating as core faculty, the students, and many other aspects in this case it seems reasonable to hope this first experience may serve as the baseline for the great array of possibilities opened for studying programs in future, showing increasingly positive results. 22