Content Validity of the Wilson-Sims Falls Risk Assessment Tool To Measure Fall Risk of Psychiatric Inpatients

Transcription

1 1 Content Validity of the Wilson-Sims Falls Risk Assessment Tool To Measure Fall Risk of Psychiatric Inpatients Mary B. Billeen, PhD, RN, NEA-BC, Ann Kruszewski, PhD, RN, Kristin Sims, RNC, MSN, NEA-BC, Sondra Fettes, RNC. MSN, Steve Wilson, RN-BC Background Accidental falls are a major concern for safety in hospital inpatients, often resulting in patient injury, prolonged hospitalization, and increased cost and litigation risk for hospitals (US Department of Veterans Affairs, 2004). The Joint Commission requires health care settings to implement and evaluate a falls reduction program, which includes assessing the patient s risk of falling and the patient s risk of injury (The Joint Commission, 2009). Prevention of patient falls is a critical issue in nursing care and falls rates are used as a standard metric of nursing care quality. Psychiatric patients are at high risk for falls, because of cognitive impairments, psychotropic medications, and behavioral manifestations such as agitation and wandering (Blair and Gruman, 2005). Evidence Various risk assessment tools for hospitalized adults have been published (Kim et al., 2007; Lovallo C., Rolandi S., Rossetti A.M. & Lusignani M., 2010). However, many of these tools do not address the specific risk factors of the inpatient psychiatric population. Perell, et. al. (2001) conducted a systematic review of descriptive studies (LOE=VA) based on a thorough search of relevant literature. A full narrative was included on a number of tools and their characteristics; however, most of the tools were developed for general inpatient or nursing home populations. A review of published literature revealed only one risk assessment tool specific to hospitalized psychiatric patients (Edmonson D., Robinson S., Hughes L., 2011). This tool included nine risk factors based on review of the literature and clinical expertise. One descriptive prospective study estimated initial predictive validity for this tool. Development and Description of the Wilson-Sims Fall Risk Assessment Tool The Wilson-Sims Falls Risk Assessment Tool (WSFRAT) was developed to identify psychiatric inpatients that are at risk for falling and require specific preventive nursing actions. The setting was Oaklawn Hospital, a community hospital with Magnet recognition The phases of creating the WSFRAT were: Phase 1) A list of risk factors was generated from a retrospective analysis of psychiatric falls cases at Oaklawn hospital from the previous two years; Phase 2) Extant literature on falls risk assessment tools was summarized; Phase 3) The initial items for the WSFRAT were formulated by synthesizing local risk factors, literature on falls assessment, and existing falls risk appraisal tools; a weighting system was established by giving items based on nurses observations three possible points and items based on a review of the medical record two possible points; Phase 4) Feasibility and initial validity were examined through a 90 day trial of the WSFRAT which included embedding the WSFRAT in the electronic medical record, staff training, and monitoring use of the tool by RN s and falls incidence in the psychiatric unit. Fall rates decreased substantially following implementation of the tool, indicating that it may have usefulness for other psychiatric settings. The tool continued to be used successfully at Oaklawn Hospital in Michigan for three years. However, the tool had not yet been formally tested. The current version has 15 fall risk areas with forced-choice questions requiring respondents to rate patients risk as 0, or 1, 2, or 3 points. Two items for summarizing the scoring are followed by a Fall Risk Comments entry box for the RN to complete. A total score of 0-6 is considered Low Risk for falling and a score of 7 or above is High Risk. The highest possible score is 39.

2 2 Purpose The purpose of this research study was to estimate the content validity of the Wilson-Sims Falls Risk Assessment tool. The results of this research will lay groundwork for a future validity testing of the WSFRAT, specifically, predictive validity and inter-rater reliability. The following definitions were used in the study: Content validity is the degree to which the items in an instrument adequately represent the universe of content for the concept being measured (Polit & Beck, 2008, p. 750). -level content validity index (I- CVI) is content validity of individual items: Proportion of content experts giving item a relevance rating of 3 or4 (Polit & Beck, 2006, p. 493). Scale-level content validity index (S-CVI) is the proportion of total items on an instrument that achieved a rating of 3 or 4 by the content experts (Beck & Gable, 2001, p. 209) A psychiatric inpatient unit is a hospital unit used for 24 hr treatment of inpatients who require psychiatric care A fall is an unplanned descent by a psychiatric inpatient to the floor or other objects on the floor, with or without injury, including falls from either physiologic or environmental reasons, and excluding falls from purposeful or planned actions (Minnesota Hospital Association, 2010; US Department of Veterans Affairs, 2004). Fall risk assessment is the process of evaluating a potential fall risk is part of providing a safe therapeutic milieu for patients with mental disorders. Methods Design A concurrent naturalistic design with a survey method was used to assess content validity of the WSFRAT. Content validity is the degree to which the items in an instrument adequately represent the universe of content for the concept being measured (Polit & Beck, 2008, p. 750). Content validity is a critical element in estimating the overall validity of an instrument. An established approach to estimating content validity involves having a team of experts indicate whether each item on a scale is congruent with (or relevant to) the construct, computing the percentage of items deemed to be relevant for each expert, and then taking an average of the percentages across experts (Polit & Beck, 2006, p. 490). Lynn s (1986) judgment-quantification method for estimating content validity of item-level content validity was used for this project. Lynn s method involves judgment by a group of five to ten experts that the individual items on a scale are content valid, and that the instrument as a whole is content valid. Polit & Beck s (2006) recommendations regarding S-CVI were utilized in data analysis. They recommend a S-CVI/Ave of.90 or higher for an instrument to have adequate content validity. Sample Purposive sampling was used to select content experts, who were chosen based on their clinical expertise and experience with prevention of inpatient falls. A list of experts was generated by reviewing existing literature on falls assessment in psychiatric inpatients, speakers at recent conferences, and posting a request on a listserve sponsored by the American

3 3 Psychiatric Nursing Association. Inclusion criteria for participants were: 1) represent diverse settings, 2) masters degree or greater, 3) leader in psychiatric nursing or inpatient nursing care; 4) fall prevention expertise (e.g., instrument development; leader for falls prevention project). After approval by the appropriate Institutional Review Board, fourteen potential experts were identified from an international sample and were contacted to determine initial interest. Of these fourteen experts, twelve agreed to participate. Ten of the 12 (83%) experts provided content validity ratings for the WSFRAT. Instrument The written questionnaire consisted of: 1. An introduction with specific instructions and operational definitions for the main constructs of a psychiatric inpatient unit, a fall, and a fall risk assessment. 2. A set of 15 items to be judged on the relevance of each item for assessing fall risk for a psychiatric inpatient, 3. A set of seven open-ended questions about the clarity and completeness of items on the WSFRAT, 4. One question regarding overall comments, 5. A copy of the WSFRAT as it appears to nurses on a computer screen. Experts were asked to judge the relevance of each item in the tool on a 4-point ordinal rating scale, and to judge the completeness and clarity of the overall tool. The 4-point rating scale included: 1 = Not ; 2= Somewhat ; 3= ; and 4 = Very. Participants comments about the clarity and completeness of the instrument were solicited to determine the need to eliminate, add, or revise items on the WSFRAT. A copy of the written questionnaire is shown in the Appendix. Procedure: After Human Subjects approval was obtained by the local institutional review board surveys were mailed to the 12 experts who agreed to participate, along with two copies of the consent form (one to be retained by participant). Participants returned the consent document and the survey via pre-paid mailers. The time required to complete the survey was estimated at 30 minutes. Participants were assured of confidentiality of their data, and informed about how to contact the investigators to ask questions prior to consent and during completion of the survey. Data Analysis A content validity index (CVI) was calculated from the survey responses using the methods proposed by Lynn (1986) and Polit & Beck (2006). The CVI for each item (I-CVI) was determined by calculating the proportion of experts who rated the item as relevant or very relevant (a rating of 3 or 4 on the survey items). If all ten experts agreed, then I-CVI=1.00 (10/10=1.00). According to Lynn (1986) with ten raters, an I-CVI of less than.70 for an item indicates that the item should be revised or eliminated. The scale CVI for the entire instrument (S-CVI) was calculated using the S-CVI/Ave method (Polit & Beck, 2006). The S-CVI/Ave is computed by averaging the proportion of items rated as relevant or very relevant (a rating of 3 or 4 on the survey items). In this study, the S-CVI/Ave was calculated as the Mean I-CVI value = sum of I-CVI/15 items. This method of calculation "puts the focus on average item quality rather than on average performance by the experts" (Polit & Beck, 2006, p.493). Adequate content validity for the instrument as a whole was determined as S-CVI/Ave of.90 rather than the usual.80 based on Polit & Beck s (2006) recommendation. In addition, participants comments about completeness and the clarity of WSFRAT items were summarized in narrative format.

4 4 Results The final sample consisted of ten experts (83% response rate) including academic researchers; physicians, advanced practice nurses; and psychiatric facility administrators from the midwest, south, and east coast of the U.S. and from Australia. The distribution of expertise included: 3 psychiatric specialists; 3 fall experts; 4 both psychiatric specialists and fall experts. Table 1 displays the participants ratings for each item on the WSFRAT. Each participant s type of expertise is listed (P = Psych; F = Fall) PF = Psychiatric and Fall). For each item, the proportion of experts giving a rating of 3 or 4 is the item-level CVI (I-CVI). For example, for the item "age," the number of experts giving a rating of 3 or 4 was nine out of ten = With ten raters, there can be three ratings of 1 or 2 for each item for an acceptable I-CVI (0.70, p =.05) (Lynn, 1986). s receiving a rating of 3 or 4 by at least seven of the ten judges in this study are considered valid. The I-CVIs of the 15 items on the WSFRAT ranged from.40 to If 2 items are excluded, gender (.40) and diuretics (.60), the range of the remaining items is.70 to The S-CVI/Ave, an averaging approach for the scale as a whole, was calculated to be.85, which is lower than the recommended value of.90. Deleting two items with low I-CVIs, (Gender =.40 and Diuretics =.60) raises the S-CVI/Ave to.91 above the acceptable.90 level. Analysis of the experts comments suggested additions and minor revisions that would improve clarity of wording and consistency of choices within items. These suggestions are described under Discussion and Conclusions. Discussion and Conclusions Acceptable content validity for each item (I-CVI) was set at.70 with ten raters. Thirteen of the fifteen items achieved this standard; however, two items, (Gender and Diuretics) did not achieve acceptable I-CVI. Gender was included in the WSFRAT because the developers recognized that falls were more likely among women in their hospital setting at the time of the tool s initial development. Experts comments suggested that gender is not a clear risk factor, and they recommended additional review of the literature to determine whether there is sufficient evidence to retain the item. Given inconclusive evidence regarding gender as a risk factor in psychiatric populations, it will be removed from the Revised WSFRAT. Diuretics were included in the WSFRAT because they affect toileting frequency and fall risk. However, this item is redundant of the Elimination item. New diuretics will be added to category 3 in the Elimination item in the revised WSFRAT, and the Diuretics item will be removed from the Revised WSFRAT. Content validity addresses the construct as a whole as to whether the items on the tool, taken together, adequately represent the universe of content for the concept being measured (Polit & Beck, 2008, p. 750). Section Two of the questionnaire addressed possible omissions in Question 4, by asking the experts if there were additional items that would improve the clarity of accuracy of the tool. One expert recommended including Agitation in the Mental Status item. Because agitation is reported as a risk factor for falls in the psychiatric literature, the Mental Status item will be revised to read 1= Oriented, Uncooperative/Agitated. Two experts commented that ECT was missing from the WSFRAT. ECT has been cited in the literature as a

5 5 risk factor for falls, but it is not used in all psychiatric settings, so the item will not be added to the revised WSFRAT. Additional comments from experts provided suggestions for refining wording of items (for example generalized muscle weakness ) and for clarifying the meaning of medication categories. A set of directions and definitions will be included in the Revised WSFRAT to improve clarity of these items. The experts recommended improving consistency of choices in the Impairments item. This item will be re-titled Sensory Impairments" and choice #2 will be reworded as "Impaired proprioception/limb (amputation, prosthesis, cast, etc. Two experts commented that the Detox protocol item was unclear regarding the type of Detox (narcotics, alcohol, etc.). Definitions of Detox protocol will be included in the Revised WSFRAT to clarify this item. The experts also commented on weighting of categories within items, and the possible over-weighting of medication items. The weighting of items will be examined in subsequent study of predictive and inter-rater reliability. The content validity for the WSFRAT as a whole was calculated using an averaging approach. The S-CVI/Ave using all 15 items was.85, which is lower than the recommended value of.90. A more stringent standard than the usual recommendation of.80 was chosen as the target value based on recommendations by Polit & Beck (2006). Eliminating the two items with the lowest I-CVI (Gender, Diuretics) would raise the S-CVI/Ave to.91 above the acceptable.90 level. Another method of determining the scale-level CVI is the universal method (S-CVI/UA). Using this method, S-CVI is calculated as the proportion of items rated as 3 or 4 by ALL experts. However, Polit & Beck (2006) state that the universal agreement method for scale-level CVIs (S-CVI/UA) is overly stringent when there are many experts on the validation panel (p.495). Because the current study included ten experts, the S-CVI-UA was not used. In conclusion, acceptable content validity can be obtained by eliminating two items (Gender and Diuretics) from the WSFRAT. This study provides the groundwork for a future study of validity of the WSFRAT. The revised scale will be tested to determine inter-rater reliability among users and predictive validity for determining falls risk.

6 6 Table 1 Wilson Sims Fall Risk Assessment Tool: Level Content Validity Index Experts & Type Age Gender Mental Status 4 Physical Status 5 6 Elimination Impairments 7 Gait or Balance 8 History 9 Mood Stabi Ilizer Meds 10 Benzodiazepines Diuretics Narcotics Sedatives/ Hypnotics 14 Atypical Anti Psycotics 1P F PF P PF F F PF PF P Number in Agreement I CVI: I CVI: is the proportion of experts who rated the item as 3 or 4 S CVI/Ave: is the Mean I CVI value = the sum of I CVI/15 items =..85 before deletions Final S CVI/Ave: with gender and diuretics deleted: = 0.91 Scale: 1 = Not relevant; 2 = Somewhat relevant; 3 = ; 4 = Very I CVI= item level content validity index S CVI/Ave= scale level content validity index 15 Detox Protocol

7 7 References: Blair, E., & Gruman, C. (2005). Falls in an inpatient geriatric psychiatric population. Journal of the American Psychiatric Nurses Association. 11(6), Edmonson, D., Robinson S, & Hughes L. (2011). Development of the Edmonson psychiatric fall risk assessment tool, Journal of Psychosocial Nursing and Mental Health Services, 49(2), Kim, E.A., Mordiffi, S.Z., Bee, W.H., Devi K. & Evans D. (2007). Evaluation of three fall-risk assessment tools in an acute care setting. Journal of Advanced Nursing, 60, Lovallo, C., Rolandi, S., Rossetti, A.M. & Lusignani M. (2010). Accidental falls in hospital inpatients: evaluation of sensitivity and specificity of two risk assessment tools. Journal of Advanced Nursing, 66(3), Lynn, M.R. (1986). Determination and quantification of content validity. Nursing Research, 35, Minnesota Hospital Association (2010). Patient Safety: Safe From Falls Call to Action. Fall Rate Definitions (March 2010). Retrieved September 20, 2010 from: Nadzam, D. (nd). Preventing Patient Falls. Joint Commission Resources Patient Safety Articles. Retrieved September 20, 2010 from Perell, K. L., Nelson, A., Goldman, R. L., Luther, S. L., Lewis, N. P., & Rubenstein, L. Z. (2001). Fall risk assessment measures: An analytic review. Journal of Gerontology, 56A, M761- M766. Polit, D.F., & Beck, C.T. (2004). Nursing research: Principles and methods (7th ed.). Philadelphia: Lippincott, Williams, & Wilkins. Polit, D.F. & Beck, C.T. (2006). The content validity index: are you sure you know what s being reported? Critique and recommendations. Research in Nursing & Health, 29, Polit, D.F., & Beck, C.T. (2008). Nursing research: Generating and assessing evidence for nursing practice (8th ed.). Philadelphia: Lippincott, Williams, & Wilkins. The Joint Commission (2008) National Patient Safety Goals. Joint Commission Perspectives, 28(7), supplement, Retrieved September 20, 2010 from: US Department of Veterans Affairs Veterans Health Administration (VHA), (2004) National Center for Patient Safety Falls Toolkit. Retrieved September 20, 2010 from:

8 8 APPENDIX: Content Validity Questionnaire: Wilson-Sims Falls Risk Assessment Section One A. The items from the Wilson-Sims Falls Risk Assessment are shown in the table below in the order they appear on the screen shot view (see last page). B. Use the definitions for assessment of fall risk for a psychiatric inpatient below for your relevance determination. C. For each item, answer the question: How relevant is this item for assessing fall risk for a psychiatric inpatient? D. Indicate the relevance of each item for assessing fall risk by circling the number in the box that best matches your opinion. Definitions: A psychiatric inpatient unit is a hospital unit used for 24 hr treatment of inpatients who require psychiatric care A fall is an unplanned descent to the floor or other objects on the floor, with or without injury, by a psychiatric inpatient, including falls from either physiologic or environmental reasons, and excluding falls from purposeful or planned actions (Minnesota Hospital Association, 2010; US Department of Veterans Affairs, 2004). Fall risk assessment: The process of evaluating a potential fall risk is part of providing a safe therapeutic milieu for patients with mental disorders Very Somewhat Not Age: 0 = years 1 = years 2 = 71 or more years Gender: 0 = Male 1 = Female Mental Status: 0 = Oriented and Cooperative 1 = Oriented and Uncooperative 2 = Confused, Memory Loss, Forgets Limitations, Intoxicated Physical Status: 0 = Healthy 1 = Generalized Muscle Weakness 2 = Dizzy, vertigo, syncope, orthostatic hypotension 3 = Cachexia and Wasting Elimination: 0 = Independent and Continent 1 = Catheter, Ostomy 2. = Elimination with Assistance, Diarrhea or Incontinence 3 = Independent and Incontinent, Urgency, or Frequency Impairments: 0 = None 1 = Uncorrected visual, hearing, language, speech 2 = Limb amputation 3 = Neurological paralysis, paresthesia Gait or Balance: 0 = Able to walk/stand unassisted or fully ambulatory. 1 = Physically unable to walk/stand (but may attempt) 2. = Walks with cane 3 = Unsteady walking, standing, walker, crutches, furniture

9 9 Very Somewhat Not History of falls in past 6 months: 0 = No History 1 = Near falls or fear of falling 2 = Has fallen 1-2 times 3 = Multiple falls, more than 2 times Mood Stabilizer Medications: Benzodiazepines: Diuretics: Narcotics: Sedatives/Hypnotics: Atypical Anti Psychotics: DETOX PROTOCOL 0 = Not on Detox Protocol 7 = On Detox Protocol PLEASE CONTINUE TO SECTION TWO ON NEXT PAGE

10 10 Section Two A. Use the screen shot view of the Wilson Sims Falls Risk Assessment to answer each question below. B. For each question, please include any suggested changes with a short rationale. C. Add your overall comments at the end. 1. Are there wording changes that would make the existing items clearer? 2. Are there changes in the weights of the existing items that would make the overall score more accurate in detecting Low or High risk for falling? 3. Should the item Fall Risk? (RN clinical judgment) that allows nurses to override the computer-generated fall risk score be altered or deleted from the tool? 4. Are there additional items that would improve the clarity or accuracy of the tool? 5. Are there changes in the order of the existing items that would improve the clarity or accuracy of the tool? 6. Are there changes in the format of the tool that would improve the clarity or accuracy of the tool? 7. Are there directions for nurses about use of the tool that should be added? Overall comments:

11 11 Screen Shot View Wilson-Sims Falls Risk Assessment Oaklawn Hospital FALL RISK ASSESSMENT Patient Assessment: PSYCH ADMISSION ASSESSMENT Age 0 = = = 71 > Gender 0 = Male 1 = Female Mental Status: 0 = Oriented and Cooperative 1 = Oriented and Uncooperative 2 = Confused, Memory Loss, Forgets Limitations, Intoxicated Physical Status: 0 = Healthy 1 = Generalized Muscle Weakness 2 = Dizzy, vertigo, syncope, orthostatic hypotension 3 = Cachexia and Wasting Elimination; 0 = Independent and Continent 1 = Catheter, Ostomy 2 = Elimination with Assistance, Diarrhea or Incontinence 3 = Independent and Incontinent, Urgency, or Frequency Impairments: 0 = None 1 = Uncorrected visual, hearing, language, speech 2 = Limb amputation 3 = Neurological paralysis, paresthesia Gait or Balance: 0 = Able to walk/stand unassisted or fully ambulatory. 1 = Physically unable to walk/stand (but may attempt) 2 = Walks with cane 3 = Unsteady walking, standing, walker, crutches, furniture History of falls in past 6 months: MEDICATIONS Mood Stabilizer Medications: Benzodiazepines: Diuretics: Narcotics: Sedatives/Hypnotics: Atypical Anti Psychotics DETOX PROTOCOL 7 points if on Detox Protocol FALL RISK SCORE: 0 = No History 1 = Near falls or fear of falling 2 = Has fallen 1-2 times 3 = Multiple falls, more than 2 times 0 = Not on Detox Protocol 7 = On Detox Protocol 0.0 (NOTE: computer generates a number based on the sum of the above items) FALL RISK LEVEL: Fall Risk? (RN clinical judgment) Fall Risk Comments: Score 0-6 = Low Risk Score 7 or Above = High Risk Yes No (NOTE: This item allows the RN to use clinical judgment to override a computer-generated Fall Risk score (NOTE: RN writes comments about fall risk factors or clinical judgment here)