Summary of a review of literature on the impact of working context and support on the postgraduate research student learning experience

Transcription

1 Summary of a review of literature on the impact of working context and support on the postgraduate research student learning experience Diana Leonard Institute of Education, University of London Who is this summary for? This summary will be of interest to: senior staff involved in the development of policy related to doctoral research degree programmes, e.g. PVCs, Heads of graduate schools, faculties and departments; staff working to support doctoral researchers, for example supervisors, skills development trainers and co-ordinators, and careers advisors. Why was the review undertaken? UK Doctoral education is currently undergoing review and redefinition. Since research and knowledge production is increasingly seen as important to sustaining international economic competitiveness, there have been moves to redefine the doctorate as a training for future researchers, rather than as primarily about the production of new knowledge or individual education. This approach is also being promoted across the EU: the Lisbon Accord (2000) is promoting the establishment of a European Research Area and aims to see Europe as 'the most dynamic and competitive knowledge economy in the world by It recognised the importance of the researcher and resulted in political objectives specifically related to the role of the researchers and the need to improve their conditions and career opportunities. These changes have followed swiftly upon measures aimed at raising general standards in doctoral education in the UK, including a revised Quality Assurance Agency (QAA) code of practice for research degree programmes and inclusion in institutional audits. Changes to the traditional PhD have also been instigated by universities and include new forms of the doctorate, measures to increase numbers (particularly of international students), and new modes of delivery. Supporting bodies and organisations such as UKCGE, the UK GRAD Programme, and local graduate schools have emerged. An adequate evidence-base to inform both practice and the development of new policy is needed, and this review set out to identify the state of the research base on

2 research degree programmes, and to draw from it those elements that influence the researcher s learning experience. This review is one of five literature reviews commissioned by the Higher Education Academy into various aspects of the student learning experience. The full text for all reviews can be accessed at For further information on and links to relevant initiatives including European and UK policy related to research degree programmes, What does the review cover? The review used a systematic approach modelled on the Institute of Education s EPPI- Centre methodology ( the basis of which is described later in this summary. Literature was initially collected from a range of countries offering a UK-style doctorate, but the volume of literature required the review to be narrowed down to UK studies. The review produced a map of the empirical literature addressing the experiences of doctoral researchers in the UK an annotated bibliography on Endnote software and a searchable online database, which includes 1135 references, many from outside the UK, with full details of the search methods employed an in-depth review of a specific question relating to UK doctoral researcher experiences of the viva examination a commentary on the systematic review methodology used, with reflections on future applications. Map of the current literature The review considered literature produced from 1985 onwards. Following the Winfield report of 1987, the Social Science Research Council commenced a research programme looking at aspects of research degrees and focussing on supervision, writing skills, QA, training in quantitative methods, completion rates and times and labour market demands in the social sciences. This programme, including two studies looking beyond the social sciences, made up the majority of pre-1995 work in this area. These were based on fieldwork conducted before the main pressures on universities to improve supervision, to speed up time and rates of completion of doctorates, and to provide taught classes in methodology began to have effect. Since 1995, relatively little research on the UK doctoral experience has been funded. What research has been done has focussed on the professional doctorate, mostly addressing how to supervise and assess knowledge production, on supervision in general and more recently, on the viva. 120 out of 415 references concerning doctoral education in the UK were identified as relevant to the review topic, i.e. relating to postgraduate researcher experience. (The selection criteria for these studies are given in systematic review methodology below). In the main, these studies were conducted by: interested academics working unfunded or with small sums from institutions, mostly focussed on social sciences and education

3 funders looking at their own students. These tend to be website reports not recorded on the BEI 1 the UK Council for Graduate Education, based on postal questionnaires and usually from the Deans and faculties perspectives doctoral researchers: there are a few unpublished theses and related publications on the doctorate individuals writing accounts of their own PhD, some in response to having failed or being referred activist bodies concerned to draw attention to and change the situation of students. There is no comprehensive evidence available on the doctoral experience from the students perspective as a basis to build policy and practice: in other words the field is mainly gaps. As regards the specific concern of this review - on-course learning experience and support - there is little work on the UK, certainly as compared to the development of this field in N America. We have no benchmark from which to judge the effects of recent changes, such as the recent emphasis on transferable skills. There is much general talk of inter-disciplinarity and internationalisation in relation to the doctorate, but next to no research on whether and how disciplinary and national academic identities may be changing. We do not know the effects on all students experiences of there being now more international and more part-time students enrolled, or of supervisors each having responsibilities for more research students. We need also to know more about life after the doctorate, and the difference that undertaking doctoral research has made to peoples lives. One of the difficulties of using the process of a systematic review was that a majority of the studies were not based on any discernible theoretical framework, and the majority presented mainly qualitative data. Literature concerned with the experiences of doctoral research students in the UK tends to focus on: the PhD, rather than on other forms of doctorate the views of HEI managers, supervisors or policy makers rather than those of doctoral candidates themselves the situation in the social sciences and, especially, education working/studying context in terms of institutional provision (just over half of studies) although there is no systematic comparison across institutions or areas of the UK outcomes e.g. completion rates (over a third of studies) pedagogy (a third of studies), the majority of these on supervision peer support (a third of studies) the viva and other forms of assessment (a quarter of studies) international research students (a high proportion of more recent studies). Research is lacking on: the mode of study and funding sources comparative analyses of gender, age, race, ethnicity, social class or disability Russell Group universities, although the nature of the host university is often not mentioned experience of research training student support, with the exception of academic literacy support 1 British Education Index

4 causes of drop out. Several general themes can be drawn from the literature: for potential science PhD candidates, the working environment and nature of training provided is as important, if not more so, than the research project itself. This contrasts with the arts, humanities and social sciences where the topic, supervisor and location are key major differences in working environment, e.g. allocated space, and incidental support, e.g. from close contact with colleagues other than a supervisor, are observed across the disciplines and appear to affect outcomes the nature of supervision is one of the main factors influencing the researcher s experience of a doctorate, and several studies show a need for supervisors to be more aware of the way in which this relationship is developing, to assess expectations and discuss problems subject discipline is a major factor in submission times and rates we know little of what happens to doctoral researchers in the years following their doctorate, and almost nothing about destinations of international students Recommendations A main outcome of this review has been the identification of the general paucity of the existing literature on the working context and support for postgraduate research students in the UK. This suggests that doctoral researchers are not seen as customers to be attracted and consulted not even to the same extent as undergraduates. With increasing international competition for doctoral researchers, especially with Australia and North America, and the UK s lead role in moves to a more structured PhD within the EU, it is important that we evaluate the changes underway. We need to know better what we are moving from, in order to retain past strengths and evaluate new initiatives. The following are recommendations to the sector based on outcomes from the review: the sector should consider ways to maintain an updated literature map on doctoral education for use by researchers and policymakers. The annotated database of literature produced by this review should be used as the basis for this. commissioned research needs to recognise the diversity of UK research students and modes of study, such that initiatives aimed specific groups of researchers, e.g. based on funding source, do not have unforeseen effects on other doctoral candidates Funding Councils should recognise the importance of research into the doctorate degree in all disciplines and consider making research on the doctoral experience in all fields eligible for submission to the RAE Bibliography An online searchable database of the 415 UK references described in this review is available at A fully-annotated Endnote database with all 1135 references related to doctoral degree programmes can be found at the Academy website: in the Directory of References. Note that in the larger Endnote file, the methodology of studies included has not been appraised and the reviewers make no comment on the findings described.

5 Impact of the viva voce The literature describing the impact on UK researchers of examination of doctoral degrees by viva was reviewed in greater depth using an extension of the systematic methodology employed for the overarching literature map. This topic was identified as a major focus for existing literature and is of particular interest given that several recent initiatives aim to improve methods of assessment, of which the viva is one part. It is hoped that this analysis will feed into the emerging debate on What is a PhD?. Using the categorisation protocol developed for the systematic review, 17 of the final 120 UK references relating to the student experience were identified as specifically addressing the impact of the experience of the viva examination on doctoral researchers in the UK. (See systematic review methodology below for details of selection criteria.) In common with the general map of literature on UK experiences, the literature concerned with the impact of the viva lacks direct evidence from researchers themselves. Few studies were assessed as high quality for systematic review, due mainly to a lack of detail in describing the methodology. Some themes, however, have emerged from the analysis: there is a lack of clarity on the part of examiners, supervisors and candidates about the purposes of the viva the viva is perceived by both supervisors and candidates as an unpredictable process and difficult to prepare for mock or practice vivas are rare, but useful mismatches between the research paradigms of researcher and examiner can lead to disagreement about the quality of a thesis the attitude and personal conduct of examiners is a key factor in whether the viva is perceived as a positive experience, even among successful candidates. Experience of the viva can have knock-on effects for an individual s research career It is clear that more could be done to improve students experience of the viva and to ensure it encourages them to do further research and to publish. The in-depth review has endorsed the importance of precepts from the QAA Code of Practice for Research Degree Programmes 2, and set out in the report a list of considerations for HEIs relating to the viva. Recommendations The Higher Education Academy, or other funding bodies, should consider extending the in-depth review of the viva to other aspects of doctoral education Systematic review methodology This review is not a review in the commonly understood sense, i.e. an essay informed by the literature. Rather, it is a selection of a sub-set of the published literature using tightly-defined inclusion criteria and based on clearly defined questions. The review was undertaken alongside the EPPI-Centre at the Institute of Education (ESRC National Centre for Research Methods) which specialises in this approach. Staff involved in the process underwent three days training, and specialist software was used alongside a systematic protocol for literature collection, inclusion and categorisation. Note that this is not an EPPI-Centre review, or a 2

6 full systemic review, as time constraints did not allow for the whole range of procedures to be carried out, such as the inclusion of a peer review. Collecting and analysing the data A database of relevant literature was constructed on an Endnote file as the basis for the review. The initial search included several countries offering a British-style doctorate, i.e. South Africa, Australia, New Zealand and the US as well as the UK. Readily accessible grey literature 3 and unpublished material was included. Entries were systematically annotated with procedures and details, e.g. to indicate their possible approach i.e. a think piece and whether or not they subsequently passed the inclusion criteria. Much more material was collected than had been expected, even though time limitations meant that not all available sources could be searched. A decision was made to focus on UK material specifically relating to researchers experiences. At the end of the review period, the file contained 1135 references, the majority from the UK and Australia (although this does not accurately reflect the balance of all available material). 415 full text UK references were screened for inclusion using the protocol developed below. Inclusion was based on criteria drawn up during the first stage of the process, which included clarifying the review question and associated definitions, planning search strategies and making internal quality assurance checks to ensure consistency of approach. The given title was refined to ask the broad question: What are the effects of different elements of the learning context on doctoral researchers experiences and outcomes? Of the 415 UK references, 120 (12 of these unpublished) were found to meet the selection criteria. Post 1985 English language studies were included if they: referred to doctoral level postgraduate research studies (including the M.Res) in the UK were about experiences as students or in the short period following studies focussed on working/ studying/ researching/ domestic and/or financial context provided some kind of empirical evidence or data or a secondary analysis of data Selected references were analysed using EPPI-Reviewer software to categorise studies according to 32 selected questions looking at features such as the type of approach adopted, the outcomes forming the focus of the study, etc. This produced a broad overview of the literature and identified gaps. The question What is the impact on research students of the process of examination of a doctoral thesis by viva voce? was used to screen the 120 UK references meeting the original inclusion criteria. 17 publications or reports representing 14 studies were identified as relevant. These were analysed using a condensed method, reflecting the EPPI data extraction process, to capture in structured summaries both results, and information on strengths of methods used in the studies. Reflections on the use of the methodology Employing this method ensures that a systematic and rigorous approach is taken to assessing the state of the literature on a given topic, and offers a high level of transparency 3 grey literature includes material such as web, conference and business reports

7 for users. EPPI-Reviewer is a flexible tool, allowing input from different team members to be compared. A major advantage of using this approach is that it is possible for others to widen or continue pieces of work because procedures and sources are recorded and explanations for decisions given. The methodology is as much, or perhaps better, suited to providing maps than to answering questions, i.e. to exposing the extent and emphasis of research, gaps in knowledge, where information on a topic is held etc. Based on experience of conducting this review, the reviewers recommend that: the HE Academy should consider the EPPI-Centre methodology and particularly the EPPI-Reviewer tool as a future mechanism for literature reviews academics and researchers generally should consider the method, and it could lead in many cases to improvements in the way empirical work is described HEIs and official bodies should ensure that theses and publications (including their own publications) are recorded on BEI and made available electronically researchers should specify fully the details of their sampling frame and of the individuals in their studies and justify their methodological approach.