Quality assurance issues for a PhD by published work: a case study

Transcription

1 Quality assurance issues for a PhD by published work: a case study The author is Emeritus Professor at the University of Hertfordshire, Hatfield, UK. Keywords Quality assurance, Qualifications, Publishing Abstract Describes a number of quality assurance issues relating to the award of the degree of PhD on the basis of published work by the University of Hertfordshire which arose over a nine-year period between 1992 and Emphasises the importance of ensuring that the academic standards associated with the award of a PhD on the basis of published work are identical with those established for the traditional route to a PhD based on an approved programme of supervised research, and that the quality assurance procedures for the two routes are as similar as possible. Concludes with the view that there are quality assurance arguments for the two routes to a PhD to be merged into a single set of regulations which allow doctoral theses to be an integral mix of published and unpublished research outcomes. Electronic access The research register for this journal is available at The current issue and full text archive of this journal is available at Volume 10. Number pp. 71±78 # MCB UP Limited. ISSN DOI / Introduction The introduction in the mid-1960s of the published work route to a British PhD was a major, if somewhat controversial, innovation. The route allows the submission of a doctoral thesis based exclusively or largely on published work and is in contrast to the conventional route to a doctorate in which candidates who have followed an approved programme of supervised research submit a thesis which, as an integral component, only rarely, and certainly not compulsorily, contains published work based on the candidate s approved research programme. The new route has similarities with the regulations for a higher doctorate, the award of which is based solely on published work, and of the requirements for a thesis for a doctorate of many Northern European universities which require published work to form a large and integral part of the thesis (Wilson, 1998). The published work route to a PhD was taken up first by the University of Cambridge in 1966 but was only slowly accepted by other UK universities. A survey conducted on behalf of the UK Council for Graduate Education (UKCGE) in 1996 (Wilson, 1996) revealed that of the 73 higher education institutions in England, Wales and Northern Ireland replying to the survey, 18 pre-1992 universities, 16 post-1992 universities and three colleges of higher education offered the route. Thus one half of the institutions participating in the survey did not offer the route at that point in time. In the two-year period prior to the survey, 72 PhDs had been awarded by this route. This represents a very small proportion of all the PhDs awarded by the institutions in the same period. A notable feature of the survey was the lack of an obvious pattern to which institutions offered the route to a doctoral degree. The universities of London, Oxford and Wales did not, but the University of Cambridge was the main exploiter of the route. Since the 1996 survey, a number of additional institutions have introduced the route, but its current status in UK universities is not known. The published work route to a PhD has been criticised for its lack of a compulsory requirement for both a component of formal education and training and the supervision of the candidate by an approved academic 71

2 researcher during the research programme. Such criticism places greater emphasis on the route to the degree than on the academic quality of the successful candidate and the associated research outcomes. In this context it is pertinent to note that the recent QAA publications on postgraduate provision argue that the distinguishing feature of the PhD, in contrast to professional doctorates, is that it is assessed solely on the basis of the quality of the final thesis and the candidate s performance in the oral examination. Perhaps more significantly, some critics of the route see it as a form of concession to normal academic standards, citing the fact that in some universities eligibility for the route is restricted to members of university staff. Indeed, only a minority of the institutions responding to the UKCGE survey offered the route to graduates of the institution and/or members of staff of affiliated research-based institutions. Such severe restrictions on eligibility encourage the suspicion that the route is an easy option for a favoured minority, thereby lowering its esteem in the eyes of some of the academic community. The route was introduced into the portfolio of research awards of the University of Hertfordshire immediately after its incorporation as a university in In the period , five PhDs were awarded via the route. Since then the number of candidates submitting for a doctorate via the route has gradually increased so that between the sessions 1995/1996 and 2000/2001, 19 PhDs have been awarded. How this gradual increase in the popularity of the route reflects changes in its popularity in other institutions is not known as no statistics are published nationally. This paper is based on the experience gained in the promotion of the route over this nine-year period. Strategy Following incorporation, the University Academic Board judged the published work route to be an appropriate alternative to the traditional route to a PhD but realised that for it to be held in the same esteem by graduates and their employers it was essential that the university demonstrated the equivalence of the academic standards 72 associated with awards by the two routes. As there have been no published reports on quality assurance issues relating to the award of PhD by this route, the regulations were framed so as to be as close as possible with those for the traditional route. The decision was also taken to appoint the most highly experienced examiners at the PhD level as possible to ensure the academic calibre of the graduates. A handbook explaining the university regulations and procedures for the operation of the route was made available to potential applicants, their employers, assessors and examiners. The route was offered to staff and graduates of the university and its precedent colleges and to staff of six research-based institutions in the vicinity of the university with which the university had long established research links. Oral presentations on the regulations, policies and procedures associated with the new route were made to university staff and staff of the collaborative research institutions. Quality assurance: general issues The university regulations governing the award of a PhD by the two routes have in common the requirements that successful candidates should have: conducted a coherent programme of research which made a significant original and independent contribution to the knowledge base of the research area; demonstrated an understanding of the research methodologies of the research area; demonstrated a knowledge of the current literature of the research area; presented a thesis in accordance with the university regulations; defended the thesis in an oral examination to the satisfaction of the appointed examiners. The regulations for the new route were approved by, and became the responsibility of, the University Research Degrees Committee (URDC) and it put in place appropriate quality assurance procedures. These remained operative until 1998 when revised regulations were introduced to reflect the experience gained over the previous six years. Major changes were made to the supervision arrangements, prima facie stage

3 and examination outcomes. For the first time the opportunity was taken to allow the award of an MPhil, but to date the degree has not been awarded. Published work was interpreted by the university regulations as research outcomes which were in the public domain and accessible by conventional search mechanisms. Candidates wishing to cite publications available only in electronic form had to satisfy the university that the publications were in a final stable form. Sponsored research available only to the sponsors or on restricted access was not allowed to be used in doctoral submissions, but this remains a debatable issue, particularly as many conventional theses are placed on restricted access by the awarding universities for intellectual property rights reasons. Quality assurance: specific issues Admission stage Applicants for admission to a doctoral degree by the published work route were selected with as great a care as those embarking on the traditional route to a doctorate. An initial assessment was made of the number, quality and coherence of the applicant s publications and judgements made on the academic potential of the applicant to make a doctoral submission. The majority of the early candidates were research staff of local pharmaceutical companies who had been involved in research programmes in the areas of chemistry, biochemistry or pharmacology. In no case had the submission of the research for a doctoral degree been foreseen, but rather it was an after event and part of a staff development strategy for a small group of staff who were regarded as potential future managers. All had been members of a multidisciplinary team led by experienced and qualified researchers and all had had the benefit of generous staff development opportunities, which had developed the personal and interpersonal generic research skills of the individuals to at least the standard sought in conventional doctoral candidates. Invariably, the publications were in the most prestigious journals. More recent candidates have been members of the university academic or research staff, a 73 number of whom had planned their research with a doctoral submission in view. One candidate was a graduate of the institution. Most were in mid-career and wished to become more involved in research activities, including the supervision of research degree students. A broad range of research areas has been represented including art therapy, business, computing, environmental management, medical economics, nursing audit and social work. Potential candidates were encouraged to make an informal preliminary enquiry about the possibility of registering for a degree via the new route as this enabled individuals whose publications were clearly not coherent, or whose research area did not coincide with the expertise available as possible supervisors, to be discouraged from making a formal application for admission. These initial discussions with the candidate were conducted either by the director of research degrees or the research leader in the appropriate university department. Once it was agreed that it would be appropriate to do so, the candidate then made a formal application for admission to the department which administered the application in the standard way. Discussions with the candidate during the admission stage most commonly centred on the questions of the number and coherence of the publications needed for a successful submission, and on the issue of establishing the candidate s contribution to multiauthored works. Initially, there was no simple advice which could be given to the numbers question, but for most of the early scientific submissions a minimum of six was recommended quite arbitrarily, but this number was often exceeded, in some cases quite significantly. With progressive feedback from assessors and examiners, numbers in excess of ten publications were subsequently positively discouraged. Some of the more recent successful submissions have been based on as few as three published works. It is now evident that a candidate s proposed portfolio of publications must be judged on its individual merits and that there is no maximum or minimum number which must be met. The coherence of the publications is in many ways a more fundamental issue as it

4 is important that the selected publications represent a linked series of research studies. The 1992 regulations required candidates to be committed to a specific portfolio of publications at the time of their formal application for admission to the university. With experience, the URDC came to realise that this was an unnecessary restriction on the progressive guidance which could be given to candidates and the requirement was eventually dropped. The 1998 review of the regulations allowed candidates to be admitted in advance of the publication of the outcome of all their research, as this enabled the candidates to benefit from advice given by the supervisors and that gained from assessors during the prima facie stage. All the research on which the submission was based had to be published by the time the thesis was submitted for examination. One of the key issues influencing the decision to admit a particular candidate was the availability of an appropriately qualified and experienced supervisor. Since none of the university s researchers had previous experience of supervising doctoral submissions by this route, and as it was inevitable that such experience would only be gained slowly, the URDC took the decision to restrict the appointment of supervisors to members of the committee. The committee is a group of 21 active researchers from across the spectrum of the university s research community. Since the committee would, as of routine, see all applications submitted via the new route, it was felt that committee members were most likely to gain familiarity with the key issues involved in the guidance of candidates using the route and hence be best able to disseminate good practice. The committee agreed that in cases where the appointed supervisor felt a need for additional specific subject expertise, then a second supervisor, but not necessarily a member of the committee, could be appointed. This need arose in less than one third of registered candidates. To distinguish the role of the supervisor from that of the principal supervisor of a traditional research degree student, the title of academic supervisor was introduced. The policy of restricting the appointment of supervisors to members of the URDC was undoubtedly beneficial for the first few years of the operation of the new 74 route to a doctorate. It ensured that consistent and relevant advice was given to candidates and that the committee quickly gained the confidence to administer the registrations. At the time of the 1998 review of the regulations, however, it was felt appropriate to relax the policy and currently two supervisors are appointed to each candidate, chosen by heads of departments using the same selection criteria as are used for the appointment of supervisors for traditional research students. The admission stage has commonly lasted several months, but adherence to admission guidelines made known to the candidate at the outset has ensured that inappropriate admission decisions have been avoided. Candidates are counselled that their admission will involve subsequent rigorous assessment of their published works and of their ownership of them before they will be eligible to make a formal submission of a thesis and submit to an oral examination. Prima facie stage Following their acceptance of an offer of admission to the university and their enrolment as a research degree student, the candidate has to prepare for the prima facie stage of the evaluation of their proposed submission. Under the terms of the 1992 regulations this involved the preparation of a 5,000-word critical appraisal of their agreed cited publications and the submission of the appraisal together with the list of publications and other supporting documentation, including the documented evidence of their contribution to multi-authored works, to a group of two or three independent assessors appointed by the URDC. Assessors were chosen for their expertise in the candidate s research area and for their experience of examining PhD candidates. They were all external to the university and were paid an appropriate fee. They were required to evaluate the proposed submission and to express an expert opinion as to whether or not the candidate had established their potential to make a successful doctoral submission. They were not provided with copies of the candidate s published works or expected to read them but if they wished to do so the university provided appropriate copies. The guidelines to assessors provided by the university stressed that assessors were not

5 expected to make definitive academic judgements on individual candidates but only to assess their potential. The reports submitted by the assessors to the URDC were invariably constructive and were used by the committee to make a decision on whether or not the candidate should be allowed to proceed to a final submission. Adverse reports resulted in the candidate being required to withdraw their registration. Constructive and critical comments embodied in the assessors reports were fed back to all candidates. This format of the prime facie stage operated for six years, but the 1998 regulations introduced a modified procedure which has many similarities with the transfer process (referred to as a progression process in the university) for traditional research students wishing to progress from an initial registration for an MPhil to that for PhD. Under the new regulations the 5,000-word critical appraisal, copies of the proposed, but not finally agreed, publications and the supporting documentation are received by an assessment team, evaluated by them on an individual basis, and the candidate is then given a short oral examination by the team. The assessment team, approved by the Faculty Research Degrees Committee, consists of one supervisor and at least one independent expert normally external to the university. A range of outcomes is available to the assessors in their recommendation to the URDC. These include requiring the candidate to revise their proposed submission, possibly by the addition or deletion of published works and with or without a resubmission for reassessment, and restricting the candidate to a submission for MPhil only. Successful candidates proceed to a doctoral submission. The main advantages of the revised procedure are considered to be the experience gained by candidates in defending their research achievements in an oral examination and of simultaneously gaining peer feedback to the benefit of their final submission. Experience has shown that candidates commonly underestimate the effort required to produce the draft critical appraisal. The importance of using the critical appraisal to establish the coherence of the published works is stressed to all candidates. Greatest difficulty has been experienced by candidates whose published works span a protracted 75 period of time, particularly in the cases of research areas which have developed rapidly in recent years. Some candidates have also experienced difficulty in establishing their individual contribution to multi-authored works. In the sciences and engineering multi-authored works are common and the challenge to individual candidates has been to present evidence of their role in the design, execution, analysis and publication of each of the works they wished to cite in their final submission. The university has required them to provide such evidence in the form of a table of the estimated percentage of the total effort required for the completion of each of the four phases for the publication of each published work. The table has to be countersigned by co-authors and ideally by other senior individuals who can vouch for the authenticity of the tabulated data, such as the candidate s line managers at the time the research presented in the work was conducted. Candidates are always advised to address any perceived unusual aspect of their individual profile in their critical appraisal and to be prepared to defend it in their oral examination. The imposition of the prima facie stage has unquestionably been an important quality assurance process for the ultimate award of a doctoral degree on the basis of published work. It has deliberately been a challenging stage en route to the candidate s final examination. Almost without exception the thoroughness of the process has been recognised and welcomed by the candidates themselves. Interestingly, of the 37 institutions identified in the UKCGE survey as offering the published work route to a PhD, only 23 reported that they too impose a comparable stage. Final submission As previously emphasised, the process leading to the award of a PhD on the basis of published work should parallel and be the same standard as that for a conventional doctoral award. Inevitably, however, the documented evidence of the quality of the candidate s research presented in theses emanating from the two routes is different. The challenge to the appointed examiners for

6 candidates via both routes is to evaluate the evidence and to reach an objective assessment of the candidate as a competent researcher, expert in their particular field and able to devise, plan and execute innovative research programmes and to modify such programmes in the light of unforeseen outcomes. To this end the oral examination should probe the important issues of the research methodology adopted, the originality, independence, coherence and recognition of the strengths and weaknesses of the completed research and the candidate s knowledge of the associated literature and general ownership of the submission. It is probable that the majority of a candidate s published works will have been subject to peer review prior to their acceptance for publication and hence that the originality of them will already have been established by the reviewers. Reports submitted by examiners, appointed for the first few candidates submitting by the published work route, identified two common issues. The first was the predictable question of the acceptable profile of a candidate s contribution to multi-authored works. The second, specific to the sciences, was the occasional lack of experimental detail allowed in many scientific publications, which prevented an adequate assessment during the oral examination of the experimental data on which conclusions in the publications were based. The latter issue was quickly resolved for later candidates by advising them to present such data in either the critical appraisal or in an accompanying appendix. The issue of individual contributions can only be addressed by providing the type of evidence suggested and by the examiners exercising their right to probe the candidate to their ultimate satisfaction or otherwise. The experience gained to date is that at the end of all the oral examinations held so far the examiners have not had cause to defer a decision on the outcome of their examination of the candidates pending the submission of additional evidence allowed by the university regulations. Examiners reports have confirmed that the content and candidate s defence of the critical appraisal was at least as vital to their assessment of individual candidates as was the quality, quantity and coherence of the 76 cited published works. The university s regulations have consistently set the size of the critical appraisal in the final thesis at 10,000 words. Many candidates have exceeded this but in no case has this led to criticism by the examiner. Most commonly, candidates have elected to present their appraisal in two parts. The first has been used to review the relevant literature at the time of the commencement of their research, and the second and larger part to place the published works in the contemporary literature, the first part being placed at the beginning of the thesis, the second immediately following the published works. It is interesting to note that the UKCGE survey identified considerable variation in the recommended size of the critical appraisal, ranging in the case of four universities from one not being required at all, through 1,000-2,000 words to 10,000 words. Perhaps more surprisingly, some institutions require no oral examination and in others the holding of such an examination is at the discretion of the examiners. The university has exercised the policy of allowing assessors for the prima facie stage to be appointed examiners in appropriate cases. It has been rare to appoint examiners who have prior experience of examining candidates for a doctorate by the published work route and experience has shown that in many cases they have benefited from being involved in both the prima facie and final examination stages. However, it has been recognised that this policy is potentially open to criticism, and accordingly, the 1998 regulations require that at least one of the appointed examiners should not have been involved in the prime facie stage. In all cases at least two and not more than three examiners are appointed, not more than one of whom may be a member of the university. Concluding remarks The experience to date within the University of Hertfordshire of the operation of the regulations for the award of the degree of PhD on the basis of published work has been entirely positive. Feedback from assessors, examiners, candidates and their employers

7 has been exceedingly supportive and has endorsed the university s decision on the value of the route to a doctorate for candidates who have not had the opportunity to study for the degree by the conventional route. Anxieties that the route represented an easy option to a doctorate have certainly been dispelled and candidates will readily testify that their experience was challenging. One candidate who was co-author of an impressive portfolio of publications was failed at the final examination stage. The examination revealed that the candidate had made a valuable contribution to the execution of the published research but had not been able to contextualise it within the current literature or identify its future direction to the satisfaction of the examiners. While this failure was a personal tragedy for the individual concerned, it clearly demonstrates that to possess a portfolio of published works does not in itself guarantee the award of a doctoral degree, the other quality assurance requirements built into the submission and examination processes weighing heavily in the opinion of the appointed examiners and of the university. In spite of the apparent acceptability and increasing popularity of the published work route, questions remain to be addressed nationally in respect of both it and the conventional route to a PhD. Publication is the final stage of the whole research process and it can be legitimately argued that gaining experience of the publication process should be an essential and integral part of the training of all researchers. However, the two routes to a PhD adopt extreme positions on the importance attached to publication. The published work route requires doctorate candidates to have published all of their research outcomes as a condition of their acceptance for registration and eventual examination for PhD. The conventional route, in contrast, seldom, if ever in UK universities, requires candidates to have published the results of their research as a prerequisite for examination. Most full time research students do make presentations at courses and conferences during their training, but a substantial proportion have not made a full publication by the time of their examination. Most commonly, publication takes place after the examination and degree 77 award. In such cases judgements are made by examiners only on the potential of the candidate s research for publication. Even when publication has occurred, the published work is most frequently incorporated in the thesis either as an appendix or as a simple insertion in a wallet at the end of the thesis. It is difficult to reconcile these extreme regulatory requirements for a common award. Experience at the University of Hertfordshire has shown that the examination process for the published work route places considerable importance on the issue of the candidate s contribution to multi-authored publications. This in turn raises questions of what exactly constitutes an acceptable profile of contributions to the various stages of research for the award of a doctorate. As an extreme example, could a senior researcher who has managed a research team and been primarily responsible for the design, analysis and publication of a series of pieces of research but who has not personally carried out much of the data gathering, be registered as a candidate for the award of a PhD? Candidates for examination for a PhD via the conventional route commonly make a declaration to the effect that the work embodied in the thesis is the result of their own research except where indicated to the contrary, but they are seldom questioned about the quantitative contribution made by their supervisor(s) to the various stages of their research programme, including the preparation of the thesis. Supervisors vary considerably in the amount of guidance and direction they give to their students research programmes and they frequently appear as co-authors in subsequent publications. These anomalies in the regulatory requirements of the two routes to a PhD identify the need for the examination demands and expectations associated with the two routes to be more transparently harmonised. To this end, the decision in principle has been taken by the University of Hertfordshire to amalgamate the regulations governing the two routes into a single set of regulations which will allow theses to contain an integral mix of published and unpublished work. A single

8 set of regulations should help to guarantee that all candidates submitting a thesis for a doctoral degree are assessed solely on the basis of the quality of their thesis and their contribution to the research outcomes embodied in it, and on their performance in the oral examination, both as envisaged by current QAA recommendations. References Wilson, K. (1996), The Award of the Degree of PhD on the Basis of Published Work in the UK, UK Council for Graduate Education, London. Wilson, K. (1998), The Status of Published Work in Submissions for Doctoral Degrees in European Universities, UK Council for Graduate Education, London. 78