Welcome. Dr Lena Low - Executive Director Corporate & Surveyor Workforce

Transcription

1 Welcome Dr Lena Low - Executive Director Corporate & Surveyor Workforce

2 My webinar is a combination of practical and academic perspectives. The practical perspective is related to my vocation while the academic is from my PhD thesis. The foundation of this webinar is The Australian Council on Healthcare Standards (ACHS) headquartered in Sydney a not for profit organisation, 3 rd oldest accreditation agency accredited by ISQua. ACHS was founded 40 years ago and currently operates in Australia and 18 other countries where it accredits >1,700 health care organisations (private and public sector hospitals, day procedure hospitals and centres, corporate health services, prisons, methadone clinics, drug and alcohol services, office based surgeries, mental health and dental facilities.)

3

4 One of my portfolios is management of our 400+ peer review surveyors who are a key component of our accreditation program. Our philosophy is that the accreditation survey should be an educational and positive developmental experience for both the organisation and surveyor. ACHS has a collaborative peer-review approach between surveyors and the organisation under review. A key role of the surveyor workforce is to conduct on-site reviews of the quality and safety aspects of the health care organisation against a set of prescribed standards. Our surveyors are from multiple disciplines including nursing, allied health, administration and medicine and are the main source of my PhD data.

5 The academic research highlights the importance of reliability and accuracy in the survey process and the lack of research in this area: The issues and influences that impact on surveyors during the survey process and the closely related topics of survey and inter-rater reliability remain under-researched (Greenfield & Braithwaite, 2007; Greenfield & Braithwaite, 2008a; Shaw, 2003a; Greenfield et al, 2009). Despite its importance, the topics of survey and inter-rater reliability remain largely unexplored (Greenfield et al, 2008b). Greenfield et al, (2009) in his findings challenged the belief that the process of accreditation is objective and many of the complex facets can be quantified. [1. Low L (2012). Medical clinician surveyors in the hospital accreditation process: their motivations for participating, the factors that influence them and how they deal with those influences. UNSW, Sydney Australia: published PhD thesis]

6 In order to determine techniques to achieving a credible survey outcome, I first had to identify what influences and issues impact on accreditation survey reliability. My study therefore had to incorporate these major areas. Two of the major areas investigated in my thesis: The influences and issues facing surveyors in the hospital based survey process The techniques they adopt to deal with these influences. [1. Low L (2012). Medical clinician surveyors in the hospital accreditation process: their motivations for participating, the factors that influence them and how they deal with those influences. UNSW, Sydney Australia: published PhD thesis]

7 Research Area Motivation(s) for becoming a surveyor Research Questions What personally motivates you to be a surveyor? What does your organisation gain from your surveying experience? What are the benefits of accreditation for health care? Issues and influences that might impact the surveyor whilst conducting the survey and have implications for survey reliability Methods for dealing with, or mitigating against, the issues and influences What are the issues and influences that exist at survey and may challenge survey reliability? Are surveyors aware of these issues and influences and how effective and consistent are they in identifying them? What implications do the issues and influences have for survey reliability during the accreditation survey process? What solutions are surveyors able to suggest to mitigate these issues and influences to facilitate survey reliability? How proficient are surveyors at suggesting solutions to influences and can they demonstrate this proficiency? What characteristics of the survey process itself promote survey reliability? What implications do the suggested solutions have for survey reliability during the accreditation survey process? Can the case study exercise used to investigate this research area form a basis for future training of ACHS surveyors?

8 (Greenfield et al, 2009) identified six factors that promoted and challenged reliability: The accreditation program, including organisation documentation and surveyor accreditation Relationship of the organisation / accrediting agency / survey team preconceived expectation of an organisation s survey can be self-fulfilling. The organisation s reputation with the accrediting organisation can influence expectations, staff conduct, accreditation agency staff and survey team behaviour Accreditation agency ability (experience and management skills) to manage and draw together the parts of the complex survey process Surveyor workforce experience and credibility Management of the surveyor workforce ensuring the surveyors have common expectations, sufficient time to perform their role and are aligned to the accrediting agency Survey team conduct and coordinator role survey teams must be appropriate, utilise collaboration and team decision making. The coordinator should facilitate the team and maintain an open relationship with the organisation. Individualistic behaviour can undermine a team assessment.

9 Training of surveyors is an essential part of an accreditation agency s role and survey reliability is an important issue (Greenfield et al, 2009). Frisino (2005) lists a range of tools and techniques that accreditation agencies utilise to promote survey reliability. These include: training of surveyors on-going appraisal of their performance ensuring surveyors are independent of the survey organisation surveyors should be experienced and act according to a code of conduct and professional standards Shaw (WHO 2004), also lists the importance of surveyor selection and training but also emphasises the report production, agreement procedures and the independence of the accreditation award process as important factors to attaining objectivity and reliability. [1. Low L (2012). Medical clinician surveyors in the hospital accreditation process: their motivations for participating, the factors that influence them and how they deal with those influences. UNSW, Sydney Australia: published PhD thesis]

10 Data were obtained from interviews of surveyors and a case study exercise involving over 400 surveyors Research Design Stage 1 Questionnaire Stage 2 In-depth Interviews Stage 3 Case Studies For investigation in future research studies What motivates medical clinicians to become surveyors? What are the influences on surveyors during the survey process? How do surveyors deal with these influences during the survey process? Are other surveyors (from any industry) open to the same influences? [1. Low L (2012). Medical clinician surveyors in the hospital accreditation process: their motivations for participating, the factors that influence them and how they deal with those influences. UNSW, Sydney Australia: published PhD thesis]

11 Researcher formulated scenarios Scenarios provided to surveyors at ACHS surveyor development days Surveyors list influences and solutions within scenarios Research team documented surveyor behaviour observed during case study Researcher aggregated influences, solutions and behavioural observations In the Case Studies, the surveyor teams were formulated to more closely replicate a real-world surveying environment. [1. Low L (2012). Medical clinician surveyors in the hospital accreditation process: their motivations for participating, the factors that influence them and how they deal with those influences. UNSW, Sydney Australia: published PhD thesis]

12 Organisation X a health service in Australia, has received more than its fair share of attention from the media and offhand professional slights, and is the butt of jokes regarding past performances and the eccentricity of some of its staff. However, this health service has provided some excellent local assistance in the case of significant natural disasters, has a strong and verified reputation for dealing with a variety of emergency presentations. As a keen and unbiased health professional constantly using evidence based decision making tools, you have a certain level of sympathy for the health service you are about to survey.

13 % of Surveyor Teams 70% 60% 50% 40% 30% 20% 10% 0%

14 % of Surveyor Teams 60% 50% 40% 30% 20% 10% 0%

15 Surveyor s professional background, personal characteristics and independence A cohesive and collaborative survey team A diverse and appropriate mix of skills within the team The role of the survey coordinator and the importance of leadership The surveyor training and ACHS support infrastructure Planning and management of the survey process Peer review and collaborative approach to survey process

16 Attitude of staff and management Status of the organisation being surveyed Hospital reputational issues including adverse media Politics (local [hospital level], area and state / federal based politics) Preconceived opinions Accreditation program Surveyor awareness of the need to be reliable and credible

17 Communication with the organisation Professional, open, non-judgemental approach to survey Dealing with issues through an effective coordinator Preparation for the survey; be aware of issues prior to the survey commencing Utilisation of own expertise and knowledge ACHS training and support during the survey Collaboration, open Validation of discussion; team findings approach Application of the standards; evidence based findings

18 Confirmed the presence of influences and issues that challenged survey reliability and credibility Demonstrated surveyors were well aware of the possible influences and issues that could impact the accreditation result Showed that surveyors recognise the imperative of being fair and balanced

19 Highlighted the solutions surveyors have available to them Lend support to current survey processes whilst acknowledging that further research into reliability and credibility would be advantageous Supported the findings of previous research into accreditation reliability, notably Greenfield, World Health Authority, and Frisino

20 ACHS maintains surveyor reliability with a range of techniques; some of which I am about to share with you.

21 Best In Show: and the challenge of Inter-Rater Reliability Author: Dr Christine Dennis, ACHS CEO Inter-rater reliability is the most easily understood form of reliability because everyone has experienced it ( For example watching any event using judges such as Olympics ice skating or a dog show, relies on human observers maintaining a degree of consistency between observers. I remember the many years my mother (pictured) tolerated the judging of her prized corgis; the heated discussions that occurred when she didn t place, yet alone win and, the excellent judging when she won!

22 Interestingly, the Dog Show Analogy has been used to explore parallels between the treatment and assessment of school students and canines at a dog show (Hitt and Helms, 2009). This analogy-based activity was believed to better present an approach to learning that was compatible with the way people think, and help people reflect on their behaviours, attitudes and assessment biases in a parallel analogue.

23 On the day of the Dog Show Analogy, we show students [trainee teachers] a video clip of a Best in Show judging. Before the judge makes his or her decision, we stop the video and ask the students to choose a winner and provide reasons for their choices. The students then share their views on the judge s decision and are asked to infer what type of criteria the judge used to make his or her decision. Hitt and Helms (2009) stated that with reflection, practice and perseverance, it is possible to create more objective and equitable learning environments.

24 From an ACHS perspective we continue to improve the validity of assessments by keeping perceptions and biases in check. We listen to the feedback we have received from our State Advisory Committee (SAC) meetings, as well as member feedback. We contribute to research (Greenfield et al 2008, Greenfield et al 2013) in an attempt to better understand and achieve improved inter-rater reliability. We are mindful that the challenge in terms of consistency in and across assessments, is a demanding task (Greenfield et al, 2013). A challenge that it is described as being evident, as noted above, in many areas of society.

25 Surveyor Development days each year in different states and internationally (> 400 surveyors) Coordinator Development Day each year (all coordinators) Training and Updates e.g. Free education, Webinars Mandatory Quizzes elearning Online Training Specialised training modules Surveyor website & support Surveyor Newsletters

26 Mentoring Program Novice status for trainee surveyors One year probation

27

28 Peer Review Organisation Surveyor Profile Organisation Profile Previous Experience Team Mix Avoid COI Good Fit and Good Experience

29 AIM: Inter-rater Reliability Identify groups within Memberships Appoint Coordinators for the groups Reinvite surveyors from previous surveys Consistency Team Mix Organisation Fit

30

31