Co-produced and patient reported experience outcomes for Dementia: A report Author: Charlotte Wood 1
About the Health Innovation Network (HIN) South London The Health Innovation Network (HIN) South London is one of the fifteen Academic Health Science Networks (AHSNs) in England with a remit to deliver a step-change in the way the NHS identifies, develops and adopts new technologies predicated on partnership working and collaboration between the NHS, academia, the private sector and other external partners within a single AHSN context and across AHSNs. 1 We are a membership organisation, and our diverse membership includes all healthcare providers (community, acute, mental health and primary care), commissioners, local authorities and higher education providers. In all our work programmes there are strong relationships and collaboration with both patient and industry partners. The health challenges we have prioritised for our local communities include diabetes, dementia, MSK, cancer and alcohol. The patient experience workstream is a cross-cutting theme which covers both stand-alone projects such as the one addressed in this paper and also supervises patient and public involvement (PPI) activities in other workstreams. More information can be found on our website. Through our programmes, particularly the way we work with partners and member organisations, we align; training and education, clinical research, informatics and innovation to improve patient outcomes and experience. Our unique approach and membership encourages strong cross-organisational and sector working using innovative technological tools and solutions to support networking. The project has been managed by Charlotte Wood, Senior Project Manager (charlotte.wood@nhs.net) with support from a Patient Experience Expert Lead and management from a Senior Programme Manager. Copyright 2014 Health Innovation Network South London. 1 http://www.england.nhs.uk/ourwork/part-rel/ahsn/. 2
Summary Following a successful use of the Delphi process at South West London and St. George s Mental Health Trust and given the role of the Health Innovation Network to spread best practice and innovation, we decided to use it to co-produce some experience metrics for what good care looks like in dementia and use these to guide service improvement in care settings. Our findings were that not only did the Delphi process not work for people with dementia, the traditional method of surveys/questionnaires are not the best way to engage with this group of people as they find large numbers of questions overwhelming, struggle to prioritise one element of care as more important than others and can find the wording of questions to be confusing and unanswerable without a lot more explanation as to what the question means and the purpose of asking it. This is useful learning to take on board, particularly because a survey is the most common method of measuring patient experience. More work could be done to find an evaluation method that is more dementia-friendly if you want to truly give the option to these people to share what does and does not work well for them. Background The aim of this project was to work with people with dementia and their carers to identify patient experience metrics 2 for dementia services through a process of co-production. The thinking behind the project was to ensure that service improvement activity is focused on what matters to people with dementia. Therefore, we adopted a holistic approach inviting people with dementia and people caring for people with dementia to participate in this process. The plan was to use a Delphi process to gain consensus on the most important metrics to be used to measure patient experience. The objective of most Delphi applications is the reliable and creative exploration of ideas or the production of suitable information for decision making. It typically involves at least two rounds of stakeholders answering questions (usually in the form of a questionnaire) with an opportunity between rounds for changes or revisions based on having others thoughts shared. The aim is to attempt to reach a consensus in our case a consensus of what is most important to people with dementia when receiving care. 2 A performance metric is a means of measuring an organisation's behaviour and performance. 3
Literature review & engagement activities Statements generated form part of Delphi exercise Reach consensus on a set of metrics which measure what' is important to patients Use metrics to identify baseline and areas for service improvement Undertake rapid service improvement and test again Figure 1 - Diagram to show Project stages. This report explains the processes followed, our attempts at a Delphi process, our attempts at using written surveys with people with dementia, the outcomes of the project and what we have learned from the experience. We go on to describe the next steps we will take. Why take the approach to co-produce metrics? Co-production can be defined as: A way of working whereby citizens and decision makers, or people who use services, family carers and service providers work together to create a decision or service which works for them all. The approach is value driven and built on the principle that those who use a service are best placed to help design it. 3 It is different from simply getting people to participate in a project participation means being consulted while co-production means being equal partners and co-creators. 4 Person-centred care is a way of thinking and doing things that sees the people using health and social services as equal partners in planning, developing and monitoring care to make sure it meets their needs. This means putting people and their families at the centre of decisions and seeing them as experts, working alongside professionals to get the best outcome. 3 National Occupational Standards SFHMF63: Work with people and significant others to develop services to improve their mental health (via http://www.scie.org.uk/publications/guides/guide51/what-iscoproduction/defining-coproduction.asp). 4 SCIE Report 53: Towards co-production: Taking participation to the next level, Feb 2012, http://www.scie.org.uk/publications/reports/report53.asp. 4
Making sure that people are involved in and central to their care is now recognised as a key component of developing high quality healthcare so much so it is one of the seven core principles of the NHS Constitution The NHS aspires to put patients at the heart of everything it does: It should support individuals to promote and manage their own health. NHS services must reflect, and should be coordinated around and tailored to, the needs and preferences of patients, their families and their carers. Patients, with their families and carers, where appropriate, will be involved in and consulted on all decisions about their care and treatment. The NHS will actively encourage feedback from the public, patients and staff, welcome it and use it to improve its services encourage feedback from the public, patients and staff, welcome it and use it to improve its services. 5 Choosing statements on what constitutes good care A literature review: What is important to people with dementia? - Statements to help assess healthcare quality based on empirical research 6 was undertaken drawing on research evidence and cross checking whether the things that people with dementia and their carers say are important to them match with more general characterisations about what is important for person-centred care. Alongside this, an engagement exercise for service users, carers and professionals took place using crowdsourcing 7, visits to dementia cafés 8 and interviews with people with dementia. This produced a long list of statements on the topics of what and how care is provided and how people feel as a result of the care provided. The forty three statements were grouped under six headings learning about dementia, using services, dignity and empathy, involvement, ongoing support and support to maintain some independence. We refined the list of statements omitting similar statements, choosing the statements with the most supporting evidence and ensuring we had a good spread of statements across the topics 5 The NHS aspires to put patients at the heart of everything it does NHS Constitution https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/170656/nhs_constitution.pdf. 6 HIN South London commissioned The Evidence Centre to complete the Literature Review - http://www.hinsouthlondon.org/system/ckeditor_assets/attachments/40/what_is_important_to_people_with_dementia.pdf. 7 Crowdsourcing can be defined as a way of obtaining ideas, solving problems and producing things by connecting online with people that you may not otherwise know or communicate with. - http://tammyesteves.me/category/egovernance/. 8 Run by the Alzheimer s Society, dementia cafés provide a safe, comfortable and supportive environment for people with dementia and their carers to socialise. 5
and sub-headings. Thirty two statements were finally selected (see table below). 1. What care is provided 1.1 Learning about dementia I was given my diagnosis of dementia quickly My diagnosis was given to me and my family in a sensitive way I have been given enough clear information about dementia I have been given enough information to plan for the future Health professionals have been open and not withheld information from me I have had the opportunity to meet other people with dementia and their families 1.2 Using services I have been given the treatment and support that is best for me and my life I can easily get services when I need to I do not have to wait too long when I need care My family or other people important to me have enough help from health and social services I am given the privacy I need to be comfortable Health services tell me about other services in the community that could help me or my family 2. How care is provided 2.1 Dignity and empathy I am treated as a person not a number Services take account of my age, ethnicity, religion, sexuality and other things that are important to me Health and social services are designed round my needs Services treat me with dignity and respect Professionals are usually friendly and kind I feel supported and understood 2.2 Involvement I feel listened to by professionals I can be involved in decisions about my care if I want to My care is a partnership between me, my family and professionals My family or other important people have been involved as much as I would like My family and I have been involved in planning what will happen in later life 2.3 Ongoing support I trust the health and social services teams that support me There is good continuity in the care I receive over time There are good links between services so one service knows what another is doing 3. How people feel as a result of care 3.1 Support to maintain some independence We are supported so I don t feel like a burden to my family I am supported so I feel part of a community I am supported so I don t usually feel depressed or anxious I am supported to keep my identify and continue activities I enjoy I have practical support with things like nutrition, sleep and hygiene My family have been helped with grief and coping Methods of consensus Table 1. The thirty two statements chosen for the survey. We attempted to use a particular method used in health service research to help gain consensus around what constitutes good care for people with dementia. The Delphi process has been 6
described as a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem. 9 It dates back from the 1950s and its use has the same underlying rationale: to establish as objectively as possible a consensus on a complex problem, in circumstances where accurate information does not exist or is impossible to obtain economically, or inputs to conventional decision making (for example by a committee meeting face-to-face) are so subjective that they risk drowning out individuals critical judgements. 10 There are several established techniques but the typical features of a Delphi procedure are an expert panel; a series of rounds in which information is collected from panellists, analysed and fed back to them as the basis for subsequent rounds; an opportunity for individuals to revise their judgements on the basis of this feedback; and some degree of anonymity for their individual contributions. 11 We were encouraged to attempt the Delphi process from the success of the Commissioning for Quality and Innovation (CQUIN) project group at South West London and St. George s Mental Health Trust where it had been used to co-produce what is considered to be an exemplar crisis plan and good practice standards for the crisis planning process in order to meet one of the Trust s CQUIN targets to review the crisis plans of people receiving a Care Programme Approach (CPA) and improve the quality of crisis planning. An extract from the report states: The three stage process for coproducing an exemplar crisis plan and good practice standards appears to have worked well. Anecdotal feedback from participants has been very encouraging about the Trust working in this way and involving a wide range of people to coproduce this work on crisis plans. The Delphi exercise has enabled service users, carers, friends, families and mental health professionals to freely express their opinions without undue social pressures to conform from others in the group but also enabled a greater number of people to be part of the process of coproducing a solution to crisis plans. 12 For our project, we planned to include people with dementia, their carers and staff working with them in a clinical and support capacity as well as voluntary sector staff (such as Alzheimer s Society dementia café facilitators) to rate the thirty two statements based on their importance. 9 Linstone, Harold A. and Murray Turoff (eds) (1975) The Delphi Method: Techniques and Applications AddisonWesley - http://is.njit.edu/pubs/delphibook/delphibook.pdf. 10 http://www.teachingenglish.org.uk/sites/teacheng/files/delphi_technique.pdf. 11 http://www.teachingenglish.org.uk/sites/teacheng/files/delphi_technique.pdf. 12 What makes a good crisis plan? CQUIN 2013/14 South West London and St. George s Mental Health NHS Trust. Pg. 2. 7
We aimed to reach a consensus, thus where scores differed amongst the groups on particular questions, we would conduct another round of the Delphi process showing people participating their original scores on a given question in the first round and the average score of each of the other responder groups to provide them with an opportunity to rescore. Following both rounds, our aim was to generate a series of statements that were rated as the most important by all groups. These would form the basis of a smaller questionnaire (preferably with fewer than 10 statements) for people with dementia at Croydon University Hospital to take to establish a baseline for how well the hospital staff were doing on what had been judged by people with dementia and those providing care alike, as the most crucial elements of good care. Undertaking rapid service improvement: the ultimate goal of the exercise Once the key statements of what was deemed most important were tested with people with dementia, the results would show where, perhaps, the care setting was not meeting expectations. So for example, if the statement Services treat me with dignity and respect made up part of the final questionnaire and few people agreed that they were treated with dignity and respect 13 this would be an area for improvement. The way to understand what needed to change in order to gain more positive feedback would be to involve people with dementia in understanding what being treated with dignity and respect would look and feel like. This was to be done at an event bringing people with dementia and hospital staff together to work through these questions and help understand how current processes might be changed or updated to make a difference. Finally, a Plan, Do, Study, Act (PDSA) cycle would be undertaken to trial a change and assess its impact, with the results of the patient/staff event feeding into the planning stage. 13 On a Likert Scale from Strongly Agree to Strongly Disagree. 8
Figure 2 - diagram to show PDSA cycle. 14 Survey one feedback and design faults Our first survey was designed to ask people to rate the importance of statements grouped under the six headings from most to least important so if there were five statements in that section, the person taking part would have to give each a score on a scale of 1 most important to 5 least important with each statement occupying a different place on the scale from 1-5. This survey was launched online and taken to a workshop at Croydon University Hospital which was attended by carers and clinicians (see Figure 3 below). Figure 3 - Screenshot of survey showing how the statements should be marked. 14 Courtesy of the NHS Institute for Innovation and Improvement http://www.institute.nhs.uk/quality_and_service_improvement_tools/quality_and_service_improvement_tools/pla n_do_study_act.html 9
The feedback we received from people who took part in the workshop was that it would be too challenging for people with dementia to undertake a survey of this nature because of the difficulty of holding the statements in their head in order to make a judgement on their importance. Likewise, others found that way of rating to be challenging not understanding the scoring system as evidenced by some responses received online which showed people rating statements at the same point on the scale 15. The wording of the statements was reported to be confusing for the carers and clinicians we spoke to who felt the statements being worded in the first person made them unsure if they could answer the question or what perspective from which to answer. Finally, it was commented that being asked to rate statements as least important when they were all seen as important was unjust and unfair. Thirty four responders (a mixture of clinicians, carers, patient experience staff and voluntary sector workers) started the survey and twenty six completed it meaning some questions were skipped over causing incomplete returns. The first statement on each page was always found to be the most important and usually the last statement on the page was found to be the least important. You could also track the highest values on a diagonal down the page (left to right) which may imply this was the way the survey was filled in. 15 Survey had been set up slightly incorrectly which assisted in this error it should have blocked the ability to rate one statement the same as another. 10
Figure 4 - Chart showing average scores (for all stakeholder groups combined) for one section. You can see the diagonal marking as explained above. Survey two changes to design and outcomes Our second survey aimed to take on board the learning from the response to our first survey and change its design so that the wording read It is important that in front of each statement (to make the wording more universal). We asked that the statements be rated on a Likert scale of strongly agree to strongly disagree. Prior to opening up this survey to a wider audience, we decided to test it with the Alzheimer s Society and clinicians to help refine the content and wording of the questions. Both fed back that it was very unlikely that we would get anything other than a strongly agree or agree for each answer particularly given the statements were all considered to be statements of relevance and importance to people with dementia. Whilst this may be interesting in itself, it would not help narrow down the statements with which to start investigating areas of service improvement therefore we did not use this approach. 11
Survey three a different approach to identify areas for improvement We realised that we needed to take a new approach and abandon the use of the Delphi process for our third attempt at a survey (which had two versions). This version for people with dementia kept statements in the first person as we were advised it was easier to understand. The other (for people who support/interact with people with dementia) changed the wording to In my experience again asking people to rate on a Likert scale of strongly agree to strongly disagree. This was to try and ensure people were responding from their point of view rather than a generic standpoint and moved away from the idea of judging the importance of the statements. What produced a consensus of strongly disagree would show areas which should be reviewed i.e. if In my experience services treat a person with dementia with dignity and respect had all groups saying they strongly disagreed, that would be an area to address and attempt to improve. Figure 5 - Screenshot showing the design of Survey three (for carers, clinicians etc.) Again, we consulted with the Alzheimer s Society on the survey design who recognised that it may solve some of the problems of the earlier surveys (particularly stopping everything being seen as equally important) but still voiced their reservations of using a survey, particularly one with so many questions with people with dementia. They volunteered to run a facilitated session at one of their peer-support groups to test the new survey. 12
Results from the peer-support group were not at all encouraging. Thirteen people with dementia (five female and eight male) aged seventy-two to ninety-one years old took part, but the survey was only partially completed because, to quote the facilitator, the group experienced increasing irritation regarding the questions. There was trouble understanding what they meant so, for example, on the statement Health professionals have been open and not withheld information from me the group answered How would we know? Likewise they did not know if they had been given a diagnosis of dementia quickly because they had no frame of reference for how long a diagnosis should take and to the question I have been given enough information to plan for the future the Facilitator reported: The group were adamant that this question was unanswerable. There was also some worry/anxiety about answering some of the questions particularly those relating to access to services possibly because of nervousness about what the consequences would be from answering them. Figure 6 - Copy of handwritten notes from the Alzheimer's Society peer-support group facilitator showing responses to questions. The survey was not answered beyond statement twelve and only four questions were successfully answered by all participants. The feedback from the facilitator was that despite her best efforts, the group lost interest in taking the survey and its length (including time it would 13
take to complete) and format, along with the format of questions were not suitable for the group. Conclusions Despite redesigning our survey from feedback and guidance, the conclusion that we have drawn is that the use of a Delphi process with people with dementia, certainly in its traditional format is not feasible. Sharing our findings with our advisor from Innovations in Dementia as well as our contacts at The Alzheimer s Society confirmed their feelings on the use of surveys particularly those consisting of a large number of questions requiring some kind of rating process not being the best way to identify how people with dementia (or indeed any cognitive impairment) are feeling and therefore uncover ways of improving their experience of care. We did have discussions about paring down the approach to ask for the five best things and five worst things about a particular service, but it was felt that this may be too general and suffer the problem of lack of context around the questions that we experienced at the peer-support group meeting. It may also be difficult for people with dementia (at any stage) to think up examples if they are tired or stressed. Given free rein for suggestions, they may, as many people might, come up with requests for/negative points about a service which cannot necessarily be delivered by the service in question i.e. stay open/be available until nine pm seven days a week, or things non-related to the service i.e. things falling under the remit of Social Services to deliver. By narrowing down a list sufficiently to stop choice being overwhelming, you are already filtering out potential options and shaping the outcome to a survey. There has been some work done by the Alzheimer s Society to produce a survey/questionnaire in order to get people with dementia to evaluate peer-support networks. They used a simple five point scale question asking participants to indicate whether they felt positively or negatively about the theme. This question was purposively simple and a pictorial aid, acting as a visual prompt, was included. The pictorial aid was sourced from Connect (UK) 16 a charity which aims to improve the lives of people living with communication disability. 17 16 Connect ideas series (2007). Including People with Communication Disability in Stroke Research and Consultation. Connect (Communication Disability Network): London. 17 Peer Support service: Results of an evaluation of three Alzheimer's Society demonstrator sites The Alzheimer s Society. 14
Figure 7 - Pictorial aid used by the Alzheimer s Society in order to help people with dementia to answer questions on a feelings scale of very positive to very negative. What did we learn? Survey s don t work for people with dementia but this is the usual way feedback on patient experience is obtained and so further work on establishing ways to reliably gather the views of people with dementia are needed. It is also challenging for people with dementia to prioritise as everything is seen as important. Next steps We are still committed to using feedback from people with dementia to drive service improvement, so in the short term will continue to analyse feedback from people with dementia and their carers in the form of stories collected on our behalf by Patient Opinion-trained volunteers within care settings in order to identify common themes (positive and negative) and implement rapid service improvement on the back of them. However, this approach is resource intensive, and is difficult to measure change against a baseline. We still believe it is important to find a practical approach to routinely measure patient experience of people with dementia, and will be using our networks to identify areas of good practice. This report has been written to summarise our project and to spread the valuable learning. 15