The Evaluation of the Welsh Assembly Government Inequalities in Health Fund 2001-2007 Welsh Institute for Health and Social Care
This document was produced as a joint publication between WIHSC and WAG Drafted and edited by: Department for Public Health and Health Professions Welsh Assembly Government Cathays Park Cardiff CF10 3NQ Original research report prepared for the Welsh Assembly Government by: Marcus Longley Tony Beddow David Cohen Jeremy Corson Morton Warner with assistance from Christalla Pithara Welsh Institute for Health and Social Care (WIHSC) Faculty of Health, Sport and Science Glyn Taff Campus University of Glamorgan Pontypridd CF37 1DL Crown Copyright August 2009 2
The Evaluation of the Welsh Assembly Government Inequalities in Health Fund 2001-2007 Introduction 1. The Welsh Assembly Government Inequalities in Health (IiH) Fund was set up in 2001 by the Minister for Health and Social Services in order to stimulate and support local action to improve health and tackle inequalities in health, including inequalities in access to health services 1. A total of 67 projects across Wales was supported through the Fund between 2001 and 2008. The core criteria for the projects were set as follows: Action that helps to address inequalities in health and the factors that cause it, action that contributes to the National Service Framework for Coronary Heart Disease and action that is consistent with the evidence base; Action that reinforces partnership and joint working between organisations in the health, local government and/or voluntary sectors; Targeted action to help deprived communities and/or disadvantaged groups within the population; Involvement of primary/community care teams to enhance and expand their role in addressing inequalities in health. 2. Of the 67 projects, 62% were led by Local Health Boards (LHB) 2, followed by NHS Trusts (22%), GP practices (6%), Local Authorities (6%), and Local Health Promotion Teams (4%). As the Fund s priority and the focus of projects action was Coronary Heart Disease (CHD), all the projects contributed to the implementation of the National Service Framework (NSF) for CHD, mainly standards 1 and 2, but some projects also addressed standards 3 and 4 (see below). Also some projects contributed to more than one standard. Standard 1: Action to decrease risk factors for CHD (61% of projects) Standard 2: Primary care action to identify those at risk for assessment/treatment (55%) Standard 3: High quality care for everyone with an acute episode of CHD (13%) Standard 4: Identification and treatment of those with heart failure (3%) Standard 5: identification and treatment of those with atrial fibrillation (0%) 3. The total annual budget for the IiH Fund was approximately 5 million, most of which was allocated to the projects. The size of grant that projects received varied. Over half of projects fell within the region of 50,000 to 100,000 per year. Working with partners, such as local government, the voluntary sector, front-line primary care and NHS trusts, each project developed diverse action or services including: Screening and risk assessment in primary care and community settings Workplace health Lifestyle change advice Exercise 1 Further details about the IiH Fund and projects supported by it can be found at http://wales.gov.uk/topics/health/improvement/communities/fund/?lang=en. 2 Local Health Boards (LHBs) succeeded Local Health Groups (LHGs) in 2002. 3
Cardiac rehabilitation programmes. 4. IiH was funded in three phases. Following the original funding between 2001 and 2004 as a 3 year pilot programme, a two-year extension was granted in recognition of projects early achievements, and to compensate for delays at the outset due to recruitment difficulties and the establishment of protocols. A further one year funding was allocated during 2007 to enable projects to continue developing services, gather more information of results and complete their selfevaluation. 5. Two evaluation contracts were commissioned to support the IiH Fund. The first provided advice and support to individual projects on a day to day basis with their self-evaluation. The second was to undertake an overarching evaluation of the Fund in its entirety. 6. This presents the key findings from the overarching evaluation of the IiH Fund. The summary was jointly produced by the University of Glamorgan Welsh Institute for Health and Social Care (WIHSC) and Welsh Assembly Government, based on the work carried out by the WIHSC 3. Methods 7. An overarching evaluation of the IiH Fund was commissioned by the Welsh Assembly Government in October 2004 to assess how effectively the IiH Fund had been in meeting its objectives. Data was collected between 2005 and 2007. The evaluation examined the key themes by focusing iteratively on three levels: national (the original conception, design, selection of projects, initiation and subsequent management of the Fund as a whole); all the funded projects (the characteristics, performance and other parameters routinely reported to the Assembly Government by the projects); and a sample of 19 projects for a more detailed study. For each level, a variety of approaches was employed as appropriate, including individual and small group interviews; focus groups and workshops; and review of documentation. For the 19 sample projects, the evaluation team carried out three visits to each project over the course of the evaluation, using a series of interviews, focus groups and documentation reviews with local stakeholders. Findings Programme Implementation Project selection 8. The selection process for projects for funding was designed to stimulate a wide variety of different bids from a variety of agencies, within a framework of competition. The application process was successfully conducted with a light burden on bidders, and the assessment was carried out fairly. A total of 112 bids was received, assessed independently against five criteria, and as a result 67 projects were chosen. The key issues associated with the selection process were linked to attracting a diverse range of bids, providing appropriate guidance on key aspects, the time available for partners to work on bids, and assessment of risk 3 A copy of the full evaluation report can be obtained from Professor Marcus Longley, Director, Welsh Institute for Health and Social Care (mlongley@glam.ac.uk). 4
factors such as critical mass, staff recruitment and retention, or degree of organisational ownership. The selection process, as well as assessing the merits of individual bids, could have taken more account of the balance across the portfolio by objective, approach, target groups and locality, to ensure that each of the Fund s objectives was adequately met. Programme and project management 9. Performance management at the project level was usually satisfactory. Most projects met their performance milestones, albeit with some adjustment to reflect changing circumstances. Quality standards appeared to be good and financial management generally sound. However, several projects struggled with issues around clarity of objectives, staff turnover, or degree of ownership within the host organisation. All projects had staff responsible for the delivery of the services or activities intrinsic to each project who were often clinically trained or otherwise expert in facets of service delivery. Frequently project staff were also those who had inspired the bid initially. Where such staff were left unsupported in terms of financial and administrative expertise they sometimes struggled with the challenges imposed by the demands of both service delivery and project management. 10. As for the management of the Fund as a whole, a small team of Assembly Government officials maintained direct relationships with all 67 projects, and to varying degrees interacted with the projects host organisations. Easy and effective lines of communication and frequent visits to projects, and a clear willingness to offer support on various operational issues by the Assembly Government were generally appreciated by the projects. Links between the Fund and other elements of the national infrastructure such as the Cardiac Networks were established. Several mechanisms were put in place to facilitate the sharing of learning among projects. These included regular regional and national meetings, a web site, and informal and ad hoc contacts facilitated by the Assembly Government and self evaluation support contractor. 11. Issues to emerge from the evaluation included: the clarity of expectations on projects; provision of external support for projects self-evaluation in the early stages; and the reporting processes. In relation to the latter, greater feedback on project reports by the Assembly Government would have been appreciated by most projects. As for the funding duration, two extensions to the funding period caused mixed reactions. While extensions were welcomed by many projects, late changes in funding assumptions, especially the second extension, were perceived as unhelpful by some local commissioners and service providers. It would have been more helpful to projects and the NHS if the duration of the Fund had been determined earlier, either at the outset, or perhaps at the point of the first extension so that all concerned could have planned accordingly. Partnership working 12. Working across organisational boundaries was a key requirement for most projects. The organisations which led projects were quite different in nature: LHBs; Local Authorities (LAs); NHS trusts; voluntary sector organisations; GP run primary care centres; and the National Public Health Service (NPHS), through individuals who had transferred from other health bodies, e.g. health promoters who came from Health Authorities. They had varying degrees of influence over local health service delivery, and this affected their ability to create real partnerships, the naturalness with which they occurred, and with whom. 5
13. Many of the projects established reasonable networking arrangements over time as a basis for partnership formation, to address the complexity and fragmentation of the world in which they operated. These arrangements were usually arrived at by a process of trial and error, heavily dependent on the knowledge and experience of the individuals concerned. There is some evidence that the level of partnership working had increased over time as well, with projects becoming better understood, partners volunteering contributions to shared work, and an element of joint planning for the future. This process might have seen speedier progress if formal developmental programmes had been offered earlier. Potential for Generalisability and Sustainability 14. The planning for the end of a project requires an assessment of the likely success of the approach, its continuing relevance, and assumptions about future funding levels in a dynamic environment where priorities and resources are likely to change over time. The very existence of a national Fund can sometimes conflict with local priorities and limit the scope for sustainability. During the course of the IiH funding period, projects were reminded by the Assembly Government of the need to address issues of sustainability, and were encouraged to engage with relevant local stakeholders, however it was difficult to compile the exit strategy at the outset of projects for various practical reasons. Many projects would have benefited from some structured consideration of future funding at a senior level by local partners as their work progressed. 15. There are many examples of approaches or specific aspects of projects which are of general interest beyond the locality of the project, including many practical lessons learnt from applying approaches in the field, and also specific toolkits, public information literature, evaluation approaches, and data collection systems. The indications at the time of writing are that about 50 of the 67 projects have good prospects of continuing local funding for at least some of the key aspects of their work, albeit sometimes delivered in rather different ways. Approaches to Tackling Health Inequalities 16. There are many different approaches to tackling health inequalities which all have their merit nationally and regionally, and in relation to specific communities and segments of the population. The IiH Fund was not established as a comprehensive approach to tackling health inequalities. The amount of money deployed through the Fund, some 5 million per year, constitutes only about 0.1% of the total NHS allocation in Wales. Instead, it was set up to inject an element of national funding to stimulate innovation, and to support worthwhile initiatives which for a variety of reasons would not attract mainstream funding. To a varying degree, it achieved all of this. Many of the projects attempted to address health inequalities, particularly inequalities in access to services between their LA/ LHB area and Wales as well as within their LA/LHB area. 17. Selecting Coronary Heart Disease (CHD) as a route into tackling inequalities in health was fully understandable given its incidence and association with a wide range of measures of inequality. Using the NSF as a framework within which investments, actions and plans should be set was also sensible. However, the specific and exclusive links with the NSF for CHD inevitably encouraged projects with a focus on health service interventions, and less on work to tackle economic and social factors affecting health inequality. 6
National and Local Linkages 18. In terms of the involvement of primary care, as noted above, the IiH Fund supported the implementation of the NSF for CHD particularly through primary prevention, but also through cardiac rehabilitation and heart failure management. The projects also supported the implementation of the new General Medical Services contract. 19. The Fund acknowledged at every opportunity its potential links to other related policy initiatives. However, relatively little evidence of tangible mutual influence between the Fund and other Assembly Government policy initiatives was found. Despite the best efforts of the Assembly Government officials responsible for the management of the Fund, it sometimes proved difficult to convince other policy areas of the relevance of the Fund and to exploit it - for example, by finding a consistent way of utilising approaches pioneered in primary care policy or integrating the Fund s experience into the development of guidance on local strategies. In some instances, linkages that might have been expected such as with Communities First programmes were made at the national level, but not always locally. 20. Nonetheless, there was some evidence of projects contributing to meeting local priorities, such as providing a service which is integral to local efforts or providing a degree of focus and dedicated resources which would not have been created from mainstream funding, given all the other pressures on local resources. Some projects forged new, or strengthened existing, local links which may be useful in terms of future joint working, for example between LHBs and Leisure Services Departments and local community groups, between secondary and primary care, or, between GP practices. Also, many of the pilot projects provided useful experience to be taken into account for future developments. Examples include projects supporting ethnic minority groups, travellers, and a deprived housing estate, as well as approaches to primary prevention, cardiac rehabilitation and heart failure management. Effectiveness of IiH Fund in Meeting its Objectives 21. The multiple objectives of the IiH Fund can be encompassed under the overall aim of reducing inequalities in coronary health in Wales. While the Fund through funded projects has made good progress towards achieving many of its subsidiary objectives as well as implementing the NSF for CHD and contributing to the implementation of the new General Medical Service contract, the general absence of good evidence on impact makes it difficult to quantify the extent to which the overall aim has been achieved. The paucity of evidence of the effectiveness of the Fund also makes it difficult to comment on its costeffectiveness. 22. The projects needs assessments generally lacked detail in identifying inequalities. Objectives were set for activity, and although it is possible to monitor changes in deprivation indicators between LHB areas over a period of years, it would be difficult to attribute these to a particular intervention. It would be reasonable to assume some effectiveness in reducing inequalities, but it is not possible to prove. Stronger public health involvement may have helped at a local 7
level to improve needs assessment, target setting, and monitoring, and at a national level to develop a strategic approach. Learning from the IiH Fund 23. Many thousands of people across Wales, usually from deprived communities, have taken part in activities and services designed to reduce their likelihood of suffering from CHD, which would not have been possible without the resources provided by the IiH Fund 4. Although in most cases there were insufficient data to quantify the likely benefit with reasonable certainty, there is reason to believe that many people will have derived some benefit. The Fund also allowed a degree of experimentation across Wales, supporting a wide variety of different approaches to tackling inequalities. Some of this experimentation is likely to be adopted on a sustained basis. 24. A policy to reduce health inequalities often demands positive discrimination, providing a service to the most deprived which may be denied to others. This approach sometimes produces local tensions which highlight some of the complexity of the whole policy to reduce inequalities. The evaluation identified that some projects interventions, despite originally being targeted at disadvantaged groups, were quickly offered to others on an undifferentiated basis. This usually stemmed from local pressures to ensure that everyone benefits from a good service an understandable demand on the NHS. 25. Each funding approach has its respective merits and drawbacks. A combination of ear-marked funding to ensure that resources go to strategic priorities, with local, multi-stakeholder determination of priorities within that strategic theme, will often provide the optimal combination. In many ways, the IiH Fund was an ambitious programme. It supported more than 60 projects with a deliberately broad range of approaches, priorities, philosophies and modalities, with little capacity for comparing and evaluating such diversity. The Fund, while it achieved much, would have greatly benefited from a strategic framework into which it could have fitted. The production of Health, Social Care and Well-Being Strategies should ensure in the future that any such investment programme is securely linked to a process of local prioritisation. Future programmes of a similar nature should be explicitly designed to support the sort of innovation for which mainstream funding is not usually available, and it should have a more rigorous element of evaluation to ensure that sufficient evidence is amassed to demonstrate which approaches really do work best. 4 An analysis of IiH projects final reports conducted by the IiH Fund management team at the Assembly Government indicates that the Fund reached over 350,000 people in Wales during the course of the programme - around 10% of the Welsh population for an outlay of 0.1% of the NHS budget annually. 8