Doctoral assessment: process, issues and the concept of originality

Transcription

1 European Universities Association 7 th EUA-CDE Workshop Dokuz Eylül University, Izmir January 2014 Hadrian s temple, Ephesus, Izmir Theme: Outcomes of Doctoral Education Mindset, Research, Innovation Doctoral assessment: process, issues and the concept of originality Gill Clarke Department of Education, University of Oxford Vice-chair, UK Council for Graduate Education

2 Summary Evolution of the PhD as a global brand Different assessment models: are they comparable, do they affect outcomes, do differences matter? Paper on the UK PhD: examines concept of originality in doctoral assessment and its interpretation at subject level

3 Bologna Declaration and earlier German/Prussian PhD influenced development of all doctorates (17 th century) Adoption of a system of easily readable and comparable degrees (Bologna Declaration, ) PhD as a qualification was way ahead: a global brand for around a century Alexander von Humboldt References: Noble (1994:6);, QAA (2012: 31); Stanford Encyclopaedia of Philosophy [online] Wilhelm von Humboldt

4 A variety of assessment models 1 Similarities Thesis (or equivalent) common to all? Viva or defence used in all countries except Australia 2? Formal requirement for originality or contribution to knowledge Licence to become an academic practitioner? Differences Timing and nature of disclosure of final outcome to candidate Nature of the oral defence: public, private, or none Number of examiners Whether or not supervisor can be present Requirement for professional practice in some subjects Pass/fail or graded 1 Kyvik, S. (2014) Assessment procedures of Norwegian PhD theses as viewed by examiners from the USA, the UK and Sweden. Assessment & Evaluation in HE, 39:2, Group of Eight (2013) The Changing PhD: discussion paper [Online] :

5 Assessment of the PhD in the UK Judgement of thesis plus viva voce in all cases A private process no public defence At least two examiners, sometimes three Independent chair/convenor may be present Supervisor may attend with candidate s permission Length of viva: hours, depending on subject

6 Some topical issues Do doctoral graduates need more skills than those they use in research? Education, or training for employment, or both? Importance of doctorate in universities research effort and knowledge exchange? 3 3 Moreno-Navarro, J. J. (2010). New Regulation for Doctoral Studies in Spain: presentation at 3 rd annual meeting of EUA-CDE, Berlin, June [Online] Available from:

7 Assessment of the PhD: Originality and its interpretation The concept of originality in the PhD: how is it interpreted by examiners? Joint authors: Gillian Clarke and Ingrid Lunt Taylor and Francis online, recent articles: published Explores ways examiners and others interpret the concept of originality when judging candidates achievements in the final PhD exam Compares two data sets Clarke, G. and Lunt, I. (2014) The concept of originality in the PhD: how is it interpreted by examiners? Assessment and Evaluation in Higher Education. Published online

8 Introduction Structure and content Originality and how it is addressed in the literature UK and international doctoral qualification descriptors Responses to a 2007 discussion paper (QAA) Emerging data from a PhD study Conclusions

9 Literature (1) Australian Qualifications Framework Council (2013) Australian Qualifications Framework. 2 nd edition. Bourke, S. and Holbrook, A. (2013) Examining PhD and Research Masters Theses. Assessment and Evaluation in Higher Education Delamont, S., Atkinson, P. and Parry, O. (2000) The Doctoral Experience: Success and Failure in Graduate School Denicolo, P.M. (2003) Assessing the PhD: a constructive view of criteria. Quality Assurance in Education Johnston, S. (1997) Examining the examiners: An analysis of examiners reports on doctoral theses. Studies in Higher Education Lovitts, B. (2007) Making the Implicit Explicit: creating performance expectations for the dissertation.

10 Literature (2) Mullins, G. and Kiley, M. (2002) It s a PhD, not a Nobel Prize : How experienced examiners assess research theses, Studies in Higher Education Ostriker, Jeremiah P., Holland, Paul W., Kuh,Charlotte V. and Voytuk, James A.(eds) (2010) A Data-Based Assessment of Research-Doctorate programmes in the United States Quality Assurance Agency for Higher Education (2001) The framework for higher education qualifications in Scotland. Glasgow: QAA. Quality Assurance Agency for Higher Education (2008) The framework for higher education qualifications in England, Wales and Northern Ireland. Gloucester: QAA.

11 Literature (3) Tinkler, P. and Jackson, C. (2000) Examining the Doctorate: institutional policy and the PhD examination process in Britain, Studies in Higher Education Tinkler, P. and Jackson, C. (2004) The Doctoral Examination Process Trafford, V (2002) Questions in a Doctoral Viva: Views from the Inside. Paper presented at the UK Council for Graduate Education Research Degree Examining Symposium George E. Walker, Chris M. Golde, Laura Jones, Andrea Conklin Bueschel and Pat Hutchings (2008), The Formation of Scholars: Rethinking Doctoral Education for the Twenty-First Century

12 Formal guidance and criteria: FHEQ and FQHES UK doctoral qualification descriptor: Creation and interpretation of new knowledge, through original research or other advanced scholarship Systematic acquisition and understanding of a substantial body of knowledge at the forefront of an academic discipline or area of professional practice General ability to conceptualise, design and implement a project for the generation of new knowledge at the forefront of the discipline

13 2007 discussion paper about doctoral degrees Total of 72 respondents, 65 of whom (90%) answered Question 8: How do you/does your institution define originality in the context of doctoral study? Group (a): 31 (43%) of respondents provided their own definitions of originality Group (b): 16 (22%) linked originality to publishability Group (c): 13 (18%) said definitions of originality should be discipline-specific Group (d): 10 (14%) did not wish to define the concept of originality Group (e): 3 (4%) emphasised the importance of a common understanding of originality Group (f): 2 (3%) wanted a reference to originality to remain within the doctoral qualification descriptor

14 Group (a) some of the definitions of originality: a contribution to knowledge, specifically, the extent to which the candidate s work provides insights into and increases understanding of their field new knowledge/discovery of new facts arising from an individual s research or creativity the application of existing knowledge in a way that provides new insights into the subject, e.g. through using different approaches or methodology forms a distinct contribution to knowledge of the subject and affords evidence of originality by the discovery of new facts and/or by the exercise of critical power the ability to think independently, find solutions to difficulties and offer fresh insights into existing situations

15 Group (b) respondents linking originality to publishability [Originality] is a red herring. The key is publishable and we are not talking of format but content. Work that meets international standards of published research making a novel contribution to the subject such that it is likely to be publishable in an appropriate journal Other comments: the outcomes of a doctorate should include a requirement to produce work of publishable quality; it should be clear examiners are looking for thesis containing work of this quality rather than having the expectation the thesis itself should be publishable; a basic quantifier for the quality of the PhD is how many publications or papers come out of the research

16 Group (c): definitions of originality should be subject-specific Various views: What expectations and assessment criteria are used in different disciplines? Defining originality is a matter for disciplines, not institutions, because the contribution is to the body of knowledge in the relevant subject Differences between STEM and arts, humanities and social sciences: in STEM, originality often linked to publication during PhD, whereas in arts etc. more emphasis may be placed on an original research concept / intellectual originality?

17 Groups (d), (e) and (f) Group (d): institutional-level definition of originality not appropriate; originality should only be interpreted at subject level Group (e): emphasised the need for a common understanding of originality, especially for new examiners Group (f): reference to originality must remain in the FHEQ/FQHES doctoral qualification descriptor

18 Emerging data from the PhD study Title: A study of how examiners judge the achievement of PhD candidates in the final examination: perspectives, process and outcomes How do examiners in a range of subjects make judgements when they assess PhD candidates (thesis and viva)? Possible reference points used: subject custom and practice; examiners previous experience; external criteria (institutional and other)

19 The case studies Case studies were carried out on a phenomenon: the doctoral examination process. This design is based on a triangulation approach to each case study, using research methods sequentially. Case study design Stage 1: Negotiate observations and/or interviews. with participating universities. Obtain copies of assessment regulations governing the procedure, including any relevant documents used by the institution to guide examiners, candidates and independent chairs, if used. Stage 3: analyse interview transcripts and examiners reports linked to the oral examinations. Note any similarities and differences in the examiners decision-making and between university regulations and guidance. Figure 2 Triangulation Stage 2: Observe and take notes at two to three vivas at five universities. Conduct interviews with candidates, supervisors and examiners (and independent chair, if in attendance), preferably just after final PhD examinations.* Request examiners reports for relevant examinations.

20 Progress with case studies and individual interviews Candidate Examiner Int Examiner Ext Supervisor Independent Chair/ Convenor Non-case study examiners University 1 1: SS 1: SS 1: SS 2: SS - University 2 2: B, A 1: B 2: A, B 1: A 2: A, B University 3 tbi 1: SS tbi tbi - 1: MB University 4 University 5 1: E 1: MB Key A = arts MB = molecular biosciences B = biological sciences SS= social sciences E = engineering tbi = to be interviewed

21 Questions for examiners As an examiner, what attributes/ characteristics/abilities/skills are you seeking in PhD candidates? For example, what questions did you have in mind when considering the recent candidate s work (thesis or equivalent) and during the viva? Did you benchmark the person with other candidates you ve examined? To what extent, if at all, did you have in mind any external criteria (including guidance at subject level)?

22 Questions for supervisors and independent chairs How do you think examiners decide whether a candidate has or has not achieved what is necessary to be awarded a PhD (thesis and viva)? What questions do they have in mind? What attributes /characteristics /abilities/skills do you think examiners are looking for in the candidate? Do examiners benchmark the person with others they have examined? How do they make their judgement(s)?

23 Questions for candidates Before the final assessment of your thesis, and before the viva, what did you think you would have to do to be awarded a PhD? For example, on what basis did you think examiners would be making their judgements, about your thesis and about you as an individual researcher? What formal guidance, if any, are you aware of that suggests what examiners should take into account? And what attributes /characteristics/abilities/skills do you think examiners are looking for in PhD candidates?

24 Overview of responses Participants refer to the question of orginality, which is often mentioned in the context of an original contribution (Lovitts, 2007: 32) They address what originality and/or a contribution to the field means for their own subject They make it clear that the concept of original work or a contribution to knowledge is only one of the criteria examiners use to assess PhD candidates

25 Categories of responses to questions Eight groups: Originality and/or a contribution to knowledge Academic level and intellectual rigour Quality of data and its analysis Methodological approach Knowledge and understanding (of the student s own work and the field of study) Publication and publishability Candidate s ability to analyse their own work critically and to defend it Quality of thesis and ownership of the work

26 Originality/contribution to knowledge To fulfil criteria set by the university, e.g. an original contribution What is the candidate s contribution to the field and does s/he have a grasp of the body of literature? Has the candidate generated new knowledge (produced something not done before or added to the understanding in/made a contribution to the field) Is it an original contribution and does it tell a coherent story?

27 Academic level/rigour and quality of data Academic level/rigour Is the work worthy of a PhD has the candidate created something the examiners can pass? Does the candidate show intellectual rigour; does the evidence and argument stand up to challenge? Quality of data and analysis What is the quality of data and its analysis and is there enough in-depth analysis? Has the candidate carried out the right statistical analysis and have they interpreted their data correctly?

28 Methodological approach How has the candidate conceptualised the research idea and is the methodology appropriate and rigorous? Can the student defend their methodological choices and argue why they have chosen the methodology used rather than a different approach; have they considered the alternatives and does their approach withstand scrutiny? Has the student understood what they ve done and can they justify their approach?

29 Knowledge and understanding What is the candidate s understanding and awareness of the broader field and the literature? Have they produced a body of work that is integrated in the field and shows how the literature has a bearing on what they ve done? Does the candidate understand the importance of controls in their experiments and have they been repeated to check validity and reliability? Is the candidate s coverage and review of the literature sufficient and appropriate? Does the review demonstrate insight rather than just the ability to summarise? Have the research questions emerged from the literature review?

30 Publication and publishability Has the candidate already published in peerreviewed publications? Is the work publishable of publishable quality? In some subjects, expectation that the candidate will present portfolio of published papers as the thesis

31 Candidate s ability to analyse and critique own work Is the candidate aware of any weaknesses in their approach and have they considered alternative interpretations? At the viva, can the candidate engage in intellectual defence of the thesis and can s/he cope with unexpected questions? Can the candidate think analytically, draw on theory when analysing their research and does s/he bring to bear on it a wider knowledge of the field? Does the candidate have the ability to critically appraise their work within the field?

32 Quality of thesis and ownership of the work Is the student s work independent their own rather than their supervisor s, using ideas they ve largely developed themselves? Has the candidate done the work and do they acknowledge others contributions appropriately? Has the candidate been honest: do they have integrity? Have the papers cited in the thesis been read? What is the overall quality of the thesis: presentation and writing?

33 Candidates responses One candidate did not mention originality in the PhD, instead focusing on the quality of research being assessed Another said I think the only criterion I was really aware of was originality, that you had to make some substantive contribution to knowledge The third candidate confirmed s/he was aware the research had to be original because of guidance by the university, which defined originality in terms of a contribution to knowledge Oxford D Phil academic dress

34 Conclusions / questions Is originality the same as a contribution to knowledge or are they different? What is a significant contribution to knowledge? Are there degrees of originality and do they relate to the candidate s overall achievement level? How does originality / a contribution to knowledge relate to publishability? Do we agree that originality can only be defined at subject level; should we try to define the way iit is interpreted by different subjects? Do all PhD candidates who are awarded the degree meet the originality / contribution to knowledge criterion, as well as criteria?