THE JOINT FUNDING BODIES REVIEW OF RESEARCH ASSESSMENT RESPONSE FROM THE ROYAL COLLEGE OF PATHOLOGISTS

Transcription

1 THE JOINT FUNDING BODIES REVIEW OF RESEARCH ASSESSMENT RESPONSE FROM THE ROYAL COLLEGE OF PATHOLOGISTS We welcome the review of the Research Assessment Exercise and the opportunity to respond and will do so under the various headings requested. 1) We accept the need for some form of assessment exercise to guide funding. Although there are flaws in the assessment process, which this exercise will attempt to address, the main problem lies with the funding side. If there are no rewards to institutions for improvement over time and the emphasis is on balancing a fixed research funding budget then ultimately smaller units will suffer and research will become concentrated into a smaller numbers of larger units better able to withstand the vagaries of research funding. 2) In addition the apparent bias against clinical research needs to be addressed, as this has had a major impact on Medical Schools, and its effect is already being seen in Academic Pathology, both in established posts and in those coming in for training in the academic grades. Recognition must be made for the fact that clinical academics devote, by contract, 6/11 of their time on NHS work. This would reduce the volume measure, and potentially the funding but this would need to compensated for by a partnership between the DoH and the DfEE. 3) In addition to considering the methods of assessment, the review should also concentrate on who is doing the assessing and a less opaque system of panel selection is required. There is widespread belief, whether true or not, that having a member on the panel is worth a grade for that member institution, and this can only lead to a loss of confidence in a system which has already proved a costly and damaging attempt of modernisation of the University research community. Group 1; Expert Review 4) A combination of both retrospective and prospective assessments should be used, with an emphasis towards the former. This would give an indication of current and most recent performance, but as research is never static with continuing recruitment and development, an indication of changing investments in staff/infrastructure should carry some weight, in view of the of the long timescale between assessments. 5) Current performance indicators are a useful surrogate for assessment. Research expenditure, numbers of Research Assistants and PhD/MD students are a marker of activity and performance, which can be taken in conjunction with the assessment of a limited number of publications during the assessment period. However, care must be taken not to disadvantage areas of medicine that are particularly involved in clinical research, and especially the craft specialties, which cannot compete with those research groupings contain a large basic science element in association with clinical research. 6) Assessment within specialties is more valuable than between specialties in these circumstances, and more reliance on methods of evaluation such as PRISM

2 should be used. We believe it important to look globally at the spread of the different disciplines, and not allow some to wither because, currently, they do not appear strong by comparison with others. 7) The section dealing with marks of esteem (RA6) is of uncertain value in its current form. We do believe that certain marks of esteem should be acknowledged and in more senior researchers are often an indication of the national and international recognition of the group, however we believe that there should be clearer guidance as to what is required. A review of current returns indicates a remarkable variation in what researchers consider to be a mark of esteem. 8) The assessment should continue at the level of higher education institutions (HEIs), as if broken down into smaller groupings, there is potential for internal competition which is potentially destructive. The system should be designed to encourage collaborative working both within and between institutions, as in the current research climate larger groups are more likely to produce internationally competitive research, whether clinical or basic. 9) The current division between UOA 1 and 3 is artificial, and does not reflect the laboratory clinical interface that is evident in most research areas these days. Cogent arguments can be made for the incorporation of the various pathology disciplines in either clinical or laboratory research areas. While combining UOA 1 and 3 would produce a very large almost unmanageable grouping, it could continue to be broken down into specialist subject panels, along the lines of the recent exercise. The other groupings of Health Services Research, Pharmacology and Dentistry should continue to remain separate. 10) The crucial element of assessment is the composition of the Review Panel. This must not only be fair but to be seen to be fair and a transparent process of panel selection, with either neutral reviewers from abroad or a much larger representation is essential. It is also important that the panels contain sufficient representation from the NHS and from all the Medical Schools so that the apparent bias against clinical medicine in the assessment can be counter-acted. The current review of research appeared to pay too much importance to basic science, and appeared to pay little attention to the impact of research on patient care, the contribution to capacity building in the NHS through training, and the implications that enforced change will have on both service delivery and clinical research which at the academic level are clearly interlinked. Group 2; Algorithm 11) Algorithm based methods of assessment are relatively poor in discerning the quality of research, and primarily are a technical assessment of work done. They provide evidence of the areas in which people are working but not whether the work is of worthwhile quality. They tend to favour work in areas, which are popular, or newly emerging, but can adversely discriminate against either specialist or smaller areas, irrespective of their importance. 12) If this method of assessment is to be used then metrics must be more sophisticated. Impact factors alone are reasonable demonstrators for pure science, but are not a fair judgement of more applied research. Other measures that could be used include patent filings, successful knowledge transfer, time to completion of successful PhDs, and other research degrees.

3 13) It is clear that in the recent exercise, some institutions manipulated their returns considerably, particularly in the proportion of staff returned. In future exercises it should either be compulsory for all HEFCE funded staff to be included in the submission, or for the rules to be clearly enunciated in advance and the impact that this would have on funding. These criteria should be laid down before the exercise and should not be altered either during or following the process. 14) We would support the incorporation of some method of algorithm assessment, if the metrics have been thought through, as it is certainly a method that is cheaper and less time consuming and is arguably predictive but the potential for manipulation must be recognised. Group 3; Self Assessment 15) Self-assessment should only be a minor indicator of research performance, unless it is based upon attaining targets that have been previously set, and can be independently audited. The targets could be universal, or could vary between HEIs, depending on their current position, and aspirations. 16) We would see the main value in self-assessment as part of an iterative process. For example, if the RAE were to take place on a 5 to 6 yearly basis with an external outside assessment, then at 2 to 3 years, an internal process of self assessment could be used that would allow the institution to critically assess its own position in preparation for the later, external process. Group 4; Historical Ratings 17) Historical assessment has little to recommend it. It is an insensitive means of assessment, and gives little evidence of current value for money. 18) Research strengths can change quickly in any given area through investment and recruitment, and any historically based method of assessment would be grossly unfair to developing institutions. It would provide a method of assessment that would be unlikely to reward improving units, and would have enormously detrimental effects on attracting established research staff and research students to newer institutions. Basically the system would lack flexibility and would remove any competitive element in future research, disadvantaging those HEIs that are making major efforts to develop. Group 5; Cross Cutting Themes 19) Research assessment should take place on no less than a 5-year cycle, and we could envisage a slightly longer time period if an internal more iterative process of self-assessment were to take place in the interim. This would acknowledge the long gap between assessment exercises, but balances the amount of effort involved in providing an external return, and the impact that this has on the research environment within the institution. Therefore a slightly longer process with intermediate fine-tuning may appropriate. 20) The ultimate aim of the assessment should be financial reward, particularly if the dual support system of funding is to continue. This would require that funding is clearly identified prior to the exercise, taking into account potential grade drift, so that institutions that improve are not unduly penalised as was seen in many of the Medical School in the recent exercise.

4 21) Part of the aim of the exercise should be to identify those areas of research that are developing and where investment is required, and would also help direct academic training needs, both those that are newly developing, and those that more traditional areas that may need additional support. 22) Excellence in research is difficult to define, and potentially raises criticisms of elitism. In the case of medicine it is important that the impact on patient care is properly considered and that the exercise properly recognises the effective relationships between research activity, patients, the community and the NHS in general, and their importance in implementing government priorities. In particular areas of with significant deprivation and need, research may reflect this with a local bias that will always compare badly with more international based work. This however should not detract from its importance to medicine in general. 23) Research in general should provide intellectual stimulation, should look at the development of new areas of exploitation, or at least alternative ways of examining existing problems, and should feed back into the development of new technology, therapeutics etc. There should also be some cost benefit analysis measure value for money. 24) The research assessment exercise should not necessarily direct the distribution of funds between subjects, as this ignores the differential costs of research in different areas and the very different potentials for alternative funding e.g. industrial, and would not necessarily be sensitive enough to pick up those research areas that are developing and may require particular early support. 25) It should be acknowledge that there is an uneven playing field with respect to research institutes, compared with traditional HEI structures and that medicine in particular, with an emphasis on clinical research and more importantly researchers with multiple commitments, may be handicapped. However this could be taken into account in the funding, and in the assessment if the methods are properly thought through and the rules are clearly laid down prior to the exercise. 26) There is certainly therefore a case for looking at the assessment of cognate subjects differently as the output may differ markedly depending on whether the subject is basic science, clinical research, or Health Services research. There should be a more unitary approach to the assessment that should be directed by the institution but should include input from all HEFCE funded individuals. This would remove the unfairness to those researchers, who feel vulnerable if they have not been included in the institutional returns, and equally would remove the temptation to the institution to manipulate the process for its own perceived advantage i.e. the willingness to manipulate numbers returned in an attempt to balance high rating and potential funding. 27) Priorities Involved i) At the outset of process adequate funding should be identified so that improvement can be rewarded. Anything less than this will have a destructive impact on research groupings, and thereby institutions, as it will reduce the ability to attract both investigators and research students. ii) It therefore goes without saying that the rules of the exercise should be clearly enunciated, and published, prior to the exercise and that these should then not be altered during the process.

5 iii) The review process itself should be transparent, both in the methods of assessment and in the selection and performance of the panels. These panels should have a broad representation and where possible should have a proportion of neutral, potentially international, researchers, and also should have significant input from lay pubic. Group 6; Have we missed anything? 28) Some of the points have already been made. The underlying aims of the RAE need to be explicitly described at the start of the process. This should include the overall agenda. The chief aim, which is to concentrate research into larger groupings, or centres of excellence, and the corollary, that this may lead to the closure of smaller institutions should be stated. The principles should be clear and transparent and should not be change during the exercise. 29) The funding stream needs to clearly announced before hand, taking into account that there will be grade drift, and this should be calculated in such a way that institutions and units that enhance their standing will be rewarded for this. 30) There is, in addition, a particular problem for institutions that host researchers funded by other organisations e.g. CRUK, as although these play an important part in the research structure and output of the organisation, cannot be included currently in their submission. This was particularly obvious in the recent exercise, where many of the institutions around the UK that host ICRF units (now CRUK), were unable to include them as part of their returns, with clear detriment. 31) We strongly believe that there should be some sort of research assessment exercise, but that there should be a cost benefit analysis of any proposed system prior to its implementation. The process should be much less bureaucratic than previously, and will require developing methods of assessment that fit in with current data collection and practice. 32) The tensions currently present between the criteria of excellence as applied to RAE submissions and the type of work that is considered particularly important for the NHS and falls under the general headings of Clinical Research and Health Services Research, which is highly valued by the local community and the health service, needs to be resolved. The recent exercise has highlighted these problems, and the outcome has been particularly to the detriment of Medical School research which will ultimately rebound on Government priorities for the NHS and ultimately patient care.