THE EVALUATION OF DOCTORAL THESIS. A MODEL PROPOSAL

Transcription

1 e-journal of Educational Research, Assessment and Evaluation Revista ELectrónica de Investigación y EValuación Educativa THE EVALUATION OF DOCTORAL THESIS. A MODEL PROPOSAL [La evaluación de tesis doctorales. Propuesta de un modelo] by/por Article record About authors HTML format de-miguel, Mario ([email protected]) Ficha del artículo Sobre los autores Formato HTML Abstract The aim of this work is the development of a model that can be used as a framework for evaluators and PhD tribunal members, either during the previous administrative processing phase or at the actual exposition and viva on doctoral dissertations. To this end, once reviewed the existing literature and normative, a new proposal is laid out on the main aspects and criteria to be analysed and assessed from any given scientific work, thus introducing a tool that facilitates and specifies this work. Our model is intended to be an open proposal on which we would welcome any contribution and suggestion. Keywords Doctoral thesis, Doctoral dissertations evaluation, Doctoral thesis quality, doctoral dissertations evaluation model. Resumen El objetivo del presente trabajo es la elaboración de un modelo que pueda ser utilizado como marco de referencia por los evaluadores y miembros de tribunales de las tesis doctorales tanto en la fase previa a la tramitación administrativa como en el acto de exposición y defensa de la misma. Para ello, una vez revisada la literatura y la normativa vigente al respecto, efectuamos una propuesta sobre los principales aspectos y criterios que deben ser objeto de análisis y valoración en todo trabajo científico, y presentamos una herramienta que facilita y precisa esta tarea. Nuestro modelo constituye una propuesta abierta sobre la que esperamos aportaciones y sugerencias. Descriptores Tesis doctoral, evaluación de tesis doctorales, calidad de las tesis doctorales, modelo de evaluación de las tesis doctorales. Introduction One of the tasks we university professors frequently have to undertake is the assessment of written work presented- beyond formal procedures in order to ensure their quality as doctoral theses. It is not customary for the universities to provide detailed protocols so that the assignment can be performed following a series of criteria that are considered a priori fundamentals for the elaboration of the mandatory reports. The rule is to employ loosely structured forms where evaluators make judgements agreeing with their particular logic about how it should be and how he/she should present a scientific work so as to qualify as a doctoral thesis. This lack of a clear and common framework about the criteria to be incorporated when assessing a piece of work of these characteristics implies that the reports issued are imprecise and impractical when taking decisions. Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 1

2 The lack of precision resulting from this state of affairs when issuing reports about a thesis is well-known: the majority of mandatory reports for the administrative processing of theses is positive the number of rejected theses does not even amount to one percent and the vast majority of them are awarded the highest grade approximately 92% of defended theses although the quality of many is questionable (Valcárcel et al., 2002). This uncommon distribution of results in the assessments of doctoral theses shows that either the protocols implemented are not suitable, or the assessment is not carried out with the scientific rigour desired. From a technical perspective the problem stems from the lack of clear and explicit criteria that should be taken into account when making required judgments. The latest Spanish legislation about Royal Decree 1393/2007, September 29 th determines that the university establishes procedures with the aim of ensuring the quality of theses in terms of formulation and the evaluation process (art. 21, 2). It states in the same text that the doctoral thesis will be evaluated in a viva to be held publicly and the examiners will issue a report and award an overall grade to the thesis according to the following scale... (art. 21, 6, 7). The meaning of the rule is clear: the procedures and criteria to evaluate the theses constitute an internal problem of the universities that should be regulated explicitly by them [1]. To date, the regulations established for doctoral programs and theses for various universities seem to address administrative questions rather than academic ones. It is unusual for universities to establish detailed regulations concerning criteria and procedures to assess the quality of the processes involved in developing a doctoral program, such as the products it generates, i.e. doctoral theses. It is customary, though, to focus the whole evaluation of the doctoral programs on statistics and quantitative indicators about entry and exit dates without analyzing the processes and causes involved in the dropping out or the considerable delay in the progress of the students. They neither require specific criteria to be considered regarding the quality of a thesis, such as its contribution to scientific knowledge on a particular field of research, or whether it might solve practical problems of social reality. As previously stated (Valcárcel et al., 2002), the final stage of assessment focused on the quality of the thesis comprises three steps or moments that should be scrutinized, i.e.: the previous assessment that each university establishes so as to authorize the administrative processing; the subsequent assessment made at the time of public presentation and defence of the thesis; and the assessment that can be completed thereafter of the publications and applications derived, as well as awards, grants, projects and other achievements generated. Obviously, the work to be done in each of these phases has different aims, and therefore requires different mandatory forms for each type of evaluation. It is therefore necessary to clarify the different approaches to these assessments based on the data and the type of report that the evaluator is asked to produce in each case. This is the issue we intend to address in this article: to present a model that can establish a framework of tasks to be performed by the evaluators who are required to assess doctoral dissertations. Thus, we have made a revision of the relevant literature on the nature and purpose of doctoral theses in order to synthesize and structure the main parameters and criteria that define the quality of this type of work, as well as their contribution to scientific knowledge. Additionally, we have elaborated a comprehensive model of the various criteria to be included in the evaluation of the thesis, which provide objective judgments and can be deployed either fully or in part depending on the type of evaluation required in each case. Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 2

3 2.- Prior assessment of the doctoral thesis reports.- According to the regulations cited, the legal text only specifies that in the evaluation process and prior to the defence/viva, the university should guarantee advertising the final doctoral thesis so that other doctors can submit comments on the contents (art. 21.4). To date, the procedures implemented in distinct universities to carry out this assessment are very diverse. They are rarely carried out by a rigorous and independent evaluation to determine objectively the quality of the work and its contribution to scientific knowledge. Likewise, it is not a good protocol to make this assessment prior to comprehensively and objectively making pertinent decisions on your processing. If the assessment procedure is not thorough, the result is predictable: the majority of these theses evaluation ends up being more of a ritual than an authentic evaluation. It should be remarked that the administrative processing of a thesis involves the supervisor s approval to proceed with its public defence, the customary report or reports required by the Department hosting the thesis, a period of public exposure so that interested doctors can submit pertinent arguments, and finally, acceptance to proceed with the defence by the University Doctoral Committee. All these processes can be considered evaluating actions or strategies, although in practice their effectiveness is reduced to a minimum. Once the thesis is given the go ahead by the supervisor, the Doctoral Committee follows suit and the panel of examiners appointed on the supervisor s suggestion is most likely to be ratified. This means that a priori the quality guarantees of any given thesis rest on the prestige that the university has, the involved Department, the Doctoral Program studied, and especially, the recognition that as a scientist the self-same supervisor holds (López Yepes et al., 2008). Although this is normally the standard procedure, there exist different ways to proceed in other scientific fields whose interesting alternatives are worth considering because they can contribute to helping solve the problem at hand. We will focus on some of them: 1. Assessment linked to the research group or team. Some university departments demand that the entire process of thesis writing from the development of the project to the writing of the dissertation be supervised by a collegiate research group or team made up of at least three competent researchers in the field of knowledge involved. The previous evaluation is therefore conditional on a favourable report by the investigators appointed together with the supervisor to ensure the quality of the writing process as well as the quality of the results obtained, since they are acquainted first-hand with the project and the work carried out by the doctoral candidates. The advantages of this strategy are unquestionable for the doctoral candidates, since it allows for the assessment and contribution of the supervisor and the rest of the members that make up the supervising group, while also constituting a guarantee for the institution. 2. Assessment linked to a pre-reading session of the thesis. In the case of well-established research groups and institutes, once the thesis is completed, the doctoral candidates perform one same presentation for all the research specialists in the group or institute so that the student can receive feedback in the form of final thoughts, opinions, and suggestions on their work. Not only does this preview or prereading of the thesis encourage the exchange of information, but it also increases the possibilities of spreading the work in national and international scientific networks. A variation of this evaluation procedure consists of complementing the pre-reading session with the mandatory reports of two external independent reviewers / examiners. Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 3

4 3. Assessments by means of external experts reports. Recently, various universities have established as a prerequisite for a thesis acceptance to proceed that two external evaluators issue their favourable reports. Although this assessment method should be given due attention, the elements involved in its development do not guarantee its efficiency. It is necessary to rigorously reconsider some of its weak points: appointment system of the evaluators, their competence and anonymity, protocol criteria for completion of assessment, degree of interrrelatedned of the reports, - since in many occasions it ends up being more of a formality than an authentic evaluation. In short, establishing an external evaluation system cannot be enough unless measures are taken to address the possible drawbacks posed above (Valcárcel et al, 2002). 4. Assessment linked to specific criteria of quality. The fact that nowadays there is widespread access to scientific works by number and type, some university departments have established a preliminary requirement: to grant a favourable report in this initial evaluation when the doctoral candidate has published two or three works on the thesis topic in journals, or has presented papers and/or attended congresses or conferences related to the scientific field in question. Commonly, under this circumstance, neither the pre-reading of the thesis nor the revision by external experts takes place. It should be remarked, however, that such a choice, constituting itself a new concept of what a submitted thesis should be, is not very common. In fact, in some academic fields the research piece is expected to be unpublished, its diffusion being carried out once the degree is awarded (Sánchez Rodríguez, 2002; López Yepes et al, 2008). The shared purpose of the options discussed above is to ensure the minimum quality of the research that is to be defended as a doctoral thesis. Given the usual lack of response from doctors during the public exposure period established by regulation and the purely bureaucratic role of the Doctoral Committee to this end, universities should establish some kind of "filter" that may offer the guarantee needed in order for the thesis to be given the go ahead. Perhaps the option of having a preliminary evaluation by two experts from outside the university, not included in the panel that has to judge the defence, would be the most appropriate choice as long as it is done with sufficient transparency; and their evaluation is made by applying a standard protocol in which comments are made explicit in an objective and binding manner. 3.- Evaluation of the Viva and Public Defense.- The present Spanish legislation does not include any novelty about the composition of panels that are to judge the thesis in the public defense. The legal text states that the examiners that evaluate the thesis must reach an agreement employing the rules established by the university. All members must have a doctoral degree and confirmed accredited experience. In any case only two members from the University responsible for issuing the degree can take part in the viva. Furthermore, when a doctoral thesis has been selected for a European mention, the board of examiners should include an external doctor pertaining to some higher educational institution or research center from a European Union country other than Spain. As previously indicated, the universities must continue to have the final word in the selection and appointment of the members that will conform the examiners board. Thus, it is the universities s duty to establish a procedure for their appointment that offers the desired credibility and grants their independence when evaluating the thesis. To this day, since the university proposals deployed to appoint viva panels stem from those of the thesis supervisor, these panels cannot be regarded as completely independent due to the networks that have been established between Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 4

5 the investigators at various academic levels. The standard procedure of enclosing a brief CV of the intended examiners has not provided enough evidence in establishing the suitability and competence of the many members of the proposed panels. No specific regulation exists about the criteria of evaluation that must be followed by examiners in order to grant a mark. The only general specification is that the panel will issue a report and grant an overall grade to the thesis in accordance with the following scale: fail, pass, good, and excellent. The panel will award the distinction cum laudem (with honours) if the overall grade is excellent and has been reached unanimously. The standard practice is for the examiners to make an individual assessment prior to the viva according to their beliefs about the parameters that such a piece of scientific work must meet. Equally, a first assessment of the thesis is made, focusing on the points that are less clear or are open to debate according to ritual, but without questioning or analysing the work following any systematic criteria previously established. Regarding the thesis defence, a number of criteria exist which are also open to evaluation. On the one hand, various authors insist on the doctoral candidate s ability to communicate the contents of his/her work clearly and rigorously, using the appropriate language of the field of study, as well as any graphic materials so as to get the message across to the examiners and the rest of the audience present (García de la Fuente, 1994; Sierra Bravo, 1999). On the other, provided that we are in front of a scientific event, it is paramount that the candidate should be able to argue appropriately the findings obtained, any advance in the area of knowledge, and the solutions to the problems formulated. In all, the doctoral candidate must show his proficiency and scientific rigor as a researcher, which may be elicited from the answers to the questions or objections posed by the examiners; his own critical assessment of the thesis; and the effective proposals advanced about new lines of research in the field (Carreras Panchón, 1994; Desantes & López Yepes, 1996). It should be noted that the defense of a doctoral thesis constitutes a fundamental part of the academic process, whose purpose is to confirm the competence of the doctoral student. In this event, the board represents the scientific community and, as such, must judge the completed work of the doctoral candidate, make the appropriate objections or suggestions. The defense will contribute the necessary evidence so as to sanction the research contributions as scientific while allowing the candidate to show his/her suitability as a researcher. For this reason the defense cannot be regarded just as a formality, but as the public evaluation of the thesis from the scientific community, as well as part of the scientific training of the doctoral candidate. However, such a view is not upheld widely, for the defense of the thesis normally amount to a social rather than a scientific event. Neither does a precise rule about the procedure for awarding a grade to a thesis exist. Apart from the comments and criticism more or less appropriate completed by the members of the panels, the grades are not discriminating. The data gathered from another work (De Miguel, 2009) reflects an atypical distribution of grades. Since the majority of the theses get the highest distinction, this should be the target of a detailed revision. From our point of view the problem lies in the fact that the universities do not require examiners to use detailed protocols in which all of the relevant questions to evaluate a thesis are being collected. This results, in practice, in most reports being prepared in a very generic manner, avoiding the most critical aspects. Sometimes, to avoid contradictions, the report is issued once the grade has been agreed upon. Hence, it would be desirable that any decision about the criteria be formulated according to a system of scales or quantitative and qualitative ranges, provided their Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 5

6 effectiveness when deciding about the grading. In conclusion, we believe that the process of evaluating a thesis should be rethought thoroughly as regards the procedure for appointing the viva examiners so that the highest degree of suitability and independence is reached; and as the introduction of criteria of rationality in the evaluation protocols to be deployed when making the mandatory reports on the thesis at the stages of processing and defense. The use of protocols where the criteria are presented objectively through quantitative and / or qualitative scales allows for a better match between the prior assessments and the grades, correcting, thus, the atypical distribution currently showing, while facilitating the production of argued reports whenever deficiencies are detected. 4.- Assessment of publications derived from the thesis.- We have already indicated that it is necessary to carry out an assessment of the thesis subsequent to its publication with the intention of measuring its true contribution to scientific knowledge and to the resolution of practical problems. For this purpose, the contributions and applications derived from the thesis through publications, patents, protocols, presentations and projects must be monitored over time. Even though this evaluation of the impact of the thesis is not an easy task, it should be a priority, since it is the fundamental criterion considered when special awards, projects, scholarships, and other forms of recognition are granted to both the authors and supervisors of the thesis. This assessment of the thesis s contribution is also important in strengthening the quality assurance systems of doctoral studies (De Miguel, 2009). Some authors, however, believe that this impact assessment should be taken into account in the thesis defense to the extent that essential parts of the thesis have been published or accepted for publication in prestigious journals. They also believe it necessary to assess the dissemination of findings made through conference presentations, seminars, or invitations from national and foreign research centers (Sánchez Rodríguez, 2002). To this end the doctoral candidate is requested to attach his resume to his defense so that the panel can evaluate the impact that the thesis has had up to that point, and that which can be expected of in the future. An initial assessment of the impact of the thesis should therefore be made at the moment of the thesis defense, though logically this type of assessment should be extended to all theses systematically and over a determined period of time (eg. 5 years). Among the possible means of disseminating the ideas of a thesis, scientific publications, those that constitute a real contribution to the progress of scientific knowledge and technical development and that have been disseminated by an adequate type of medium within the scientific community, provide the most immediate channel. It should be noted that in order for a publication to be considered scientific, it must contribute to the advancement of knowledge within a discipline or about a social problem and be disseminated through media that is subject to anonymous and independent internal control by experts. The Scientific Committee of the publishing medium chosen journal, publishing company, guarantees in this way the quality of a work for its publication, as not all publications have the same scientific relevance. In the fields of the Social Sciences and Education, preferential value is given to contributions published in journals with a longstanding reputation, accepting as such those that occupy leading positions in databases like the Journal Citation Report of the Social Sciences Citation Index (Institute of Scientific Information, Philadelphia, PA, USA). The gauge employed by this institution to evaluate the quality of a publication is an impact factor constructed by averaging the citations Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 6

7 received by a publication in a particular journal. The impact factor is calculated by counting the number of citations a publication receives in a particular journal each year, using the two previous years to determine the impact factor of a given year. The greater this impact factor, the greater the quality of the publication (De Miguel, 1997). The use of this gauge, while widely accepted and frequently used in other scientific areas, causes difficulties when applied to the field of the Social Sciences and Education, given the peculiarities of research in this field, the diversity of existing publications, and the lack of specialized computerized databases. Though we have recently made progress in this area and now have some databases that collect publications regarding education, we must recognize that we are still far behind other scientific fields in which there is general knowledge of which publications and databases are most prestigious and what requirements a work should meet to be accepted for publication. Due to the current difficulties in using this indicator, others are used that can also offer us an estimation of the quality of a thesis. Among these, the following may be noted: the total number of works derived from the thesis (Np), the total number of citations of works derived from the thesis (Nc), appointments for work or the ratio between Nc and Np, number of significant works (Ns), number of citations of the most cited works (Cq), number of works that have an amount of citations equal or superior to a stated number (Index h of Hirsch). All these itemsallow us to evaluate, in varying degrees, the quality of a publication and can be used as much as the derivatives of a thesis. Finally, we wish to state that, while other means of measuring the impact of a thesis in its field of research exist, the most important qualifier is its contribution to the construction of knowledge that leads to intervention into and resolution of problems, in our case educational ones. It is therefore unacceptable that many theses are not intended for publication. Unfortunately, a good part of what is called gray literature in the Social Sciences and Education consists of unpublished theses. Currently, with the digital support and the resources offered by the internet, the possibilities of disseminating theses and making them accessible have greatly increased; even though this situation has also led to other problems (intellectual property, author s rights, ) that will need to be looked into (Moralejo Alvarez, 2000; Merlo Vega y Sorli Rojo, 2002; Orea, 2003). 5.- A model proposal to evaluate doctoral theses.- Once we have analyzed the different reports involved in the evaluation of a doctoral thesis we should advance the proposal for a model that could constitute a framework for the work to be carried out by the evaluators at different times or stages of the process. To this end we have developed a comprehensive model that includes the main items to be taken into account when reviewers make their judgments about the doctoral theses. Our purpose has been to set up a tool that can act, in part, as a guide including the main items that should be subject to analysis and assessment; and, secondly, a tool to capture and contrast the quantitative and qualitative assessments that are made by different evaluators. In any case, we wish to consider our model as an open proposal that each evaluator will have to adapt in each case. For the development of this model, we have used as a theoretical framework the literature about the development of doctoral theses (Alcina, 1994; Sierra, 1999; Blaxter et al, 2000, López Yepes et al, 2005, Uriz et al, 2006)), as well as some concrete proposals recently written by other researchers from similar fields (López Yepes et al 2008). To develop the procedure we have used a simple technique: pose some questions that the evaluator should formulate about the basic Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 7

8 aspects that a thesis should comply with as a scientific work, specifying in each case the criteria of evaluation that we use to deliver valuable judgments Criteria for evaluating the report on the thesis.- When an evaluator is faced with a thesis report, the questions that should logically be formulated in order to carry out its assessment are very simple and focus on four broad dimensions or sections that should be specified precisely in this type of work: the statement of the problem, the specification of the research methodology, the analysis of the results, and the assessment of the report submitted as a thesis. On each of these dimensions, an assessment should be made adopting the evaluation criteria for scientific articles themselves. A.- The problem under investigation.- 1. Questions: What is the problem under investigation? Does it satisfy the required characteristics to be considered a scientific investigation? Is it formulated in an appropriate and clear form? Are the hypothesis and objectives specified correctly? Is it original? Does it relate to the available knowledge up to the present time? Was a good review of the literature made? Is it up to date? Does the fundamental theory have internal and external consistency? Can it be proved? Is it pertinent to carry out this type of investigation at this time and in this context? Can it be included within the doctoral program and the lines of work of the research group? Is it relevant? Does it address an important problem from a scientific point of view?, 2. Evaluation criteria: clarity, originality, pertinence, and relevance of the problem. B.- Research methodology used.- 1. Questions: What research methodology is used? Is it consistent with the objectives? Is it pertinent? Is it the most appropriate at the present time and in this context? Does it correctly specify all of the steps? Does it question the use of alternative methods? How were the samples selected? Do the instruments used satisfy the necessary conditions? Are the analytical tools handled adequately? Does it present new methods or techniques? Is the methodology applied in a parsimonious form? Does it present all of the necessary information to justify the conclusions? Are adequate rates of reliability and validity presented in the results? Is the work replicable?, 2. Evaluation criteria: suitable methodology, accurate, parsimonious, and replicable. C.- Analysis of the Results.- 1. Questions: What results support the thesis? Are they presented adequately? Are they justified from the findings? Are they understandable? Is there any kind of bias? Are they significant for the development of knowledge? Are they novel? Do they reaffirm or contradict previous knowledge? Can they be useful? Are they transferable? Are they presented in a critical fashion? Does it propose other outstanding issues for investigation? Appropriate for dissemination? 2. Evaluation criteria: understandable results, significance, novelty, and usefulness. D.- Formal aspects of the report.- 1. Questions: Does the thesis satisfy the necessary conditions to be considered a scientific work? Are the questions Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 8

9 presented in a clear fashion? Is it thorough? Is it well written? Is the style appropriate? Is the data included easily understandable? Are the sources specified? Is it well cited? Does it include up-to-date references? Does it conform to the standards of scientific works?, 2. Evaluation criteria: clarity of the report, precision of the content, adequate citation, and adjusted to scientific standards Criteria for evaluating the presentation and defense of the thesis. As stated above, the examiners should evaluate the presentation and defense of the doctoral thesis at the viva. The questions and criteria to be used for evaluation are: 1. Questions: Has the content of the work been presented to the public in a clear and understandable manner? Has the candidate focused his or her presentation on the fundamental aspects of the dissertation? Have the technical devices contributed effectively to the presentation? Was the candidate open to criticism, observations, and pertinent suggestions formulated during the presentation? Has he/she answered all of the theoretical and methodological questions posed in relation to his/her work? Were the responses adequate and correct?, 2. Evaluation Criteria: Clarity in the presentation of the work and adequacy of the defense given Relative criteria about the impact of the thesis. Although we have already said that the evaluation of the thesis scientific impact should be evaluated over a period of time, at the time of the defense, a set of indicators could be adopted in order to advance prospective judgments about the contributions to scientific knowledge that can be expected from the work in question. Hence an initial assessment of the impact of the thesis is in order in these terms: 1. Questions: Has the work had any kind of projection as of yet? Have new lines and research projects been generated from the thesis? Does it provide methodologies of interest for researchers? Impact of the work on the community? Type of output publications? Presentations in conferences and seminars? Invitations received? Dissemination of the thesis among the scientific community? 2. Evaluation criteria: Screening in the field of research and impact of contributions in the scientific field. Following from the above, we have developed a number of protocols (see the annex) that present the practical dimensions and criteria which we believe the evaluators of the thesis should take into account when giving their personal assessment. The protocol specifies the criteria that we have identified as appropriate for the report prior to processing both by external evaluators and by the members of the panel appointed for this purpose (I); those relating to the exposition and defense of the thesis by the doctoral student in the public session (II); and those that can be used in a first assessment on the likely impact of the work (III), although logically the latter course should be disaggregated and nuanced in terms of the purpose that justifies the assessment post-thesis in each case. We propose a double scaled evaluation postthesis- quantitative and qualitative- revolving around the 20 established criteria we believe have facilitated the formulation of the overall thesis, the proposal to grant the qualification and preparation of mandatory reports. The protocol that is attached regarding the evaluation of the report (IV) especially the final table- may estimulate the exchange of views and dialog between the members of the Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 9

10 panel at the time of establishing a joint evaluation, determining whether or not to pass the thesis, and deciding an appropriate grade. As stated in the regulations "the board shall issue a report and the overall grade for the thesis according to the following scale: fail, pass, good and excellent. The board may grant the distinction <cum Laudem> if the overall mark is excellent and has been reached unanimously" (art. 21.7). We believe that the use of scales as presented here helps to introduce rational criteria in the evaluation of the thesis, especially when the time comes to decide the grade of the thesis. Finally, we would like to stress that the model we have presented constitutes an open protocol for the scientific community that may be subject to discussion and improvement by adding or changing certain aspects and criteria. As with all personal work, we are aware of its limitations, but we also believe that this may serve as an initial proposal for the construction of a useful and practical tool in the evaluation of theses. Employing an attached summary table may contribute effectively to the evaluation of the thesis. At any rate, it is the evaluators and the members of the viva panels who will judge whether the proposed criteria are appropriate to the extent that they facilitate their task when it comes to making observations and judgments about the thesis, and consequently, taking decisions on the appropriate grading. References Alcina, J. (1994). Aprender a investigar: métodos de trabajo para la redacción de tesis doctorales (Humanidades y Ciencias Sociales). Madrid: Compañía Literaria. Blaxter, L., Hughes, C. y Tight, M. (2000). Cómo se hace una investigación. Barcelona: Gedisa. Carreras Panchón, A. (Coord.) (1994). Guía práctica para la elaboración de un trabajo científico. Bilbao: CITA, Publicaciones y Documentación. De Miguel, M. (1997). La evaluación de la actividad investigadora del profesorado en el ámbito de las ciencias de la educación. Revista de Investigación Educativa, 15 (1), De Miguel, M. (2010). La evaluación y mejora de los estudios de doctorado. Revista de Educación, 352 (en prensa). Desantes, J. y López Yepes, J. (1996). Teoría y técnica de la investigación científica. Madrid: Síntesis. García de la Fuente, O. (1994). Metodología de la investigación científica. Madrid: Ediciones CEES. López Yepes, J., Fernández Bajón, M.T. y Prat, J. (2005). Las tesis doctorales: Producción, evaluación y defensa. Madrid: Fragua. López Yepes, J., Fernández Bajón, M.T., Orera, L., Sánchez Vigil, J.M., Martínez Montalvo, E., Hernández Pacheco, F., Prat, J. y Sanchez, C.M. (2008). Criterios para la evaluación de tesis doctorales. Revista General de Información y Documentación, 18, Merlo Vega, J, y Sorli Rojo, A. (2002). Bases de datos y recursos en Internet sobre tesis doctorales. Revista española de documentación científica, 25 (1), Ministerio de Educación y Ciencia. Real Decreto 1393/2007, de 29 de octubre, por el que se establece la ordenación de las enseñanzas universitarias oficiales. (BOE, 30 de Octubre 2007, núm. 260, ) Moralejo Álvarez, M. (2000). Las tesis doctorales de las universidades españolas: control bibliográfico y acceso. Revista General de Documentación e Información, 10 (1), Orea, L. (2003). La edición digital de tesis doctorales: hacia la resolución de problemas de accesibilidad. Revista interamericana de Bibliotecología, 26 (1), Sierra Bravo, R. (1999). Tesis doctorales y trabajos de investigación científica (5ª edición). Madrid: Paraninfo. Sánchez Rodríguez, E. (Coord). (2002). Estudios de tercer ciclo en España. Programas de doctorado, tesis doctorales, indi- Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 10

11 cadores de calidad del tercer ciclo. MEC/ Dirección General de Universidades, Programa Estudios y Análisis. Úriz, Mª J., Ballestero, A., Viscarret, J.J. y Ursúa, N. (2006) Metodología para la investigación. Pamplona: Ediciones, Eunate. Valcárcel, M. (Direct.) (2002). El doctorado en las universidades españolas. Situación actual y propuestas de mejora. Cordoba: MEC/Dirección General de Universidades, Programa Estudios y Análisis. Notes [1] Recently (in March 2010) the Spanish Universities Secretary General, Ministry of Education, circulated a draft of the Royal Decree Project to regulate official Doctoral Education. Although the text is not yet final, the specific regulations regarding the evaluation and defence of theses do not introduce substantial changes, with the exception of the proposal suggesting that the distinction cum laude not exceed, annually, 20% of the appointed and successful theses. ANNEX 1 I.-PROTOCOL FOR EVALUATING THE REPORT OF THE THESIS A: The problem under investigation CRITERIA QUESTIONS ASSESSMENTS Clarity of the problem What is the problem under research? Does it satisfy the required characteristics to be considered a scientific investigation? Is it formulated in an appropriate and clear form? Are the hypotheses and objectives specified correctly? Does it have a good theoretical basis or foundation?, Originality Is it original? Which sources of data and information are resorted to? Was a good review of the literature made? Is it up to date? Does it relate to the current knowledge up to the present time? Are new methods or procedures presented?, Pertinence Is it pertinent to carry out this type of investigation at the present time and in this context? Does it register along the lines of the doctoral program and/or the research group?, Is it relevant? Is it considered an important problem to resolve in the field of Relevance education? Is it adequately justified? Will it contribute useful knowledge? Does it respond to important current demands? Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 11

12 B: Methods of research used I.-PROTOCOL FOR EVALUATING THE REPORT OF THE THESIS (continuation) CRITERIA QUESTIONS ASSESSMENT What research methods are used? Does a connection between the theoretical framework and the methodology exist? Is it Adecuate adecuately justified? Is it written in the most coherent way for methodology what it is trying to say? Does it introduce some new contribution?, Are the design, the variables, the samples, the procedures for collecting data, and the data analysis techniques presented in a detailed and sufficient Correct specification manner? Are all the steps specified in correct fashion? Do the instruments have reasonable levels of reliabilty and validity?, Is the methodology applied in a parsimonious form? Are the Parsimonious application Replicable method investigative procedures specified step by step? Is the process of data analysis thorough? Does it present all of the necessary data to justify the conclusions?, Is it replicable? Can the design and procedure that were used be replicated with other projects? Are there other works that exist that can be used as a reply? Are these works cited?, Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 12

13 C: Analysis of the Results I.-PROTOCOL FOR EVALUATING THE REPORT OF THE THESIS (continuation) CRITERIA QUESTIONS ASSESSMENT What results and conclusions support the thesis? Are they presented in an adecuate fashion? Are they understandable? Are they warranted from the obtained data? Do Understandable results they answer the proposed hypotheses and objectives? Is there a disagreement with the conclusions?, Are they important for the development of knowledge? Do they reaffirm or contradict previous Significance knowledge? Do they contribute to the resolution of specific problems?,. From a theoretical and methodological point of view, are they novel? Are they presented in a Originality critical fashion? What are the limitations of the study? Does it propose other outstanding issues for investigation?, Can they be useful? For whom will they be useful? Does it present any guarantee in this regard? Should it Usefulness be published? Does it carry out proposals and recommendations useful for immediate application?, Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 13

14 D: Formal Aspects of the Report I.-PROTOCOL FOR EVALUATING THE REPORT OF THE THESIS (continuation) CRITERIA QUESTIONS ASSESSMENT Clarity Are the contents presented in a clear form? Is the report well written? Is the style appropriate? Does it have an adequate structure? Are the data, the tables, and the graphs that are included easily understandable? Precision Is there internal consistency between the theory and the practice? Is the report thorough? Is it elaborated in a systematic form? Is scientific language used?, Documentation Is it well cited? Does it specify the sources used? Does it include up-to-date references? Are the quotations and references pertinent? Do they match the text?, Does the report have an adequately formal presentation? Does the thesis satisfy the necessary Adjusted to conditions to be considered scientific standards a scientific work? Does the writing conform to the norms of scientific works? Is it publishable?, E: Evaluation of the Presentation and Public Defense II.- PROTOCOL FOR EVALUATING THE PRESENTATION AND DEFENSE OF THE THESIS CRITERIA QUESTIONS ASSESSMENT Clarity of the presentation Adequacy of the defense Has the content of the work been presented to the public in a clear and understandable manner? Have the fundamental aspects of the work been stressed? Did the technical means used contribute effectively to the presentation that was made?, Has the doctoral candidate expressed openness to criticism and relevant comments made by the panel? Has he/she answered all of the theoretical and methodological questions posed in relation to his/her work? Were the responses adequate and correct?, Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 14

15 F: Evaluation of the Impact III.- PROTOCOL FOR EVALUATING THE IMPACT OF THE THESIS CRITERIA QUESTIONS ASSESSMENT Screening in the field of research Are new lines of research on the problem under investigation proposed? Does it contribute interesting methodologies for researchers? Has the thesis had an impact in other projects or investigations? Have new valuable models or tools been generated to be implemented in other projects?, Contributions to scientific knowledge Is there any sort of dissemination of the work at a scientific level: publication, seminars, communications,? Is there any expectation of publishing? Has any kind of model or patent been generated? What impact can be expected from the contributions advanced by the thesis?, Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 15

16 INFORMATION ABOUT THE DISSERTATION SUMMARY IV. PROTOCOL FOR THE REPORTS ON THE THESIS Title Author Supervisor Department University INFORMATION Name ABOUT THE Institution EVALUATOR Appointment 3 DIMENSIONS TO BE EVALUATE A: The problem / hypothesis under investigation SCORE ASSESSMENT B: Methodology of the research used C: Analysis of the Results ASSESSMENT OF THE THESIS D: Formal aspects of the report E: Evaluation of the presentation and public defense F: Evaluation of the Impact OVERALL GRADE OBSERVATIONS AND COMMENTS: Date and Signature: ANNEX NOTES [2] 1= Fail 2= Pass 3= Good 4= Excellent [3] Indicate whether the Doctoral Thesis is presented from within or outside the Department/University. Whether you are a member of the proposed Panel, and in that case, the Panel position you occupy Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 16

17 ABOUT THE AUTHORS / SOBRE LOS AUTORES de-miguel, Mario ([email protected]): Chair of Research Methods in Education at the University of Oviedo (Spain). It is integrated in the scientific area of 'Research Methods and Diagnostics in Education' (MIDE). His mailing address is: Departamento de Ciencias de la Educación. C/ Aniceto Sella s/n Oviedo (España). Find other articles by this author in Scholar Google Reference / Referencia Title / Título Authors / Autores Review / Revista ISSN Publication date / Fecha de publicación Abstract / Resumen Keywords / Descriptores Institution / Institución Publication site / Dirección Language / Idioma ARTICLE RECORD / FICHA DEL ARTÍCULO de-miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1. The evaluation of doctoral thesis. A model proposal. [La evaluación de tesis doctorales. Propuesta de un modelo]. de-miguel, Mario RELIEVE (Revista ELectrónica de Investigación y EValuación Educativa / E-Journal of Educational Research, Assessment and Evaluation), v. 16, n (Reception Date: 2009 September 16; Approval Date: 2010 April 20; Publication Date: 2010 April 20). The aim of this work is the development of a model that can be used as a framework for evaluators and PhD tribunal members, either during the previous administrative processing phase or at the actual exposition and viva on doctoral dissertations. To this end, once reviewed the existing literature and normative, a new proposal is laid out on the main aspects and criteria to be analysed and assessed from any given scientific work, thus introducing a tool that facilitates and specifies this work. Our model is intended to be an open proposal on which we would welcome any contribution and suggestion. El objetivo del presente trabajo es la elaboración de un modelo que pueda ser utilizado como marco de referencia por los evaluadores y miembros de tribunales de las tesis doctorales tanto en la fase previa a la tramitación administrativa como en el acto de exposición y defensa de la misma. Para ello, una vez revisada la literatura y la normativa vigente al respecto, efectuamos una propuesta sobre los principales aspectos y criterios que deben ser objeto de análisis y valoración en todo trabajo científico, y presentamos una herramienta que facilita y precisa esta tarea. Nuestro modelo constituye una propuesta abierta sobre la que esperamos aportaciones y sugerencias. Doctoral thesis, Doctoral dissertations evaluation, doctoral thesis quality, doctoral dissertations evaluation model. Tesis doctoral, evaluación de tesis doctorales, calidad de las tesis doctorales, modelo de evaluación de las tesis doctorales Departamento de Ciencias de la Educación, Universidad de Oviedo - (España). English and Spanish versions (Title, abstract and keywords in Spanish and English) RELIEVE Revista ELectrónica de Investigación y EValuación Educativa E-Journal of Educational Research, Assessment and Evaluation [ISSN: ] Copyright, RELIEVE. Reproduction and distribution of this articles it is authorized if the content is no modified and their origin is indicated (RELIEVE Journal, volume, number and electronic address of the document). Copyright, RELIEVE. Se autoriza la reproducción y distribución de este artículo siempre que no se modifique el contenido y se indique su origen (RELIEVE, volumen, número y dirección electrónica del documento). Revista ELectrónica de Investigación y EValuación Educativa [ ] pag. 17