What do I do with all this text? Demystifying qualitative analysis

Transcription

1 What do I do with all this text? Demystifying qualitative analysis ROSE HENNESSY, MPH PREVENTION & EVALUATION COORDINATOR WISCONSIN COALITION AGAINST SEXUAL ASSAULT Objectives Learn and practice three ways to interpret and evaluate qualitative analysis including: (1) categorizing qualitative responses into binary outcomes (2) categorizing qualitative responses into a multi-option coding system (3) a general group coding and interpretation process Participants will have the opportunity to explore questions and challenges related to qualitative data analysis and reporting options

2 Analysis Methods 101

3 Quantitative # Analysis Percentages: 40% of people agreed BEFORE training that adults can keep children safe from abuse. 75% of people agreed AFTER training that adults can keep children safe from abuse. Mean / Average: The mean score was a 85% Quantitative Analysis: The methods we use to analyze qualitative data. (numbers) Types of Qualitative txt Data (Open-ended) Qualitative Analysis: The methods we use to analyze qualitative data. (text)

4 1. When should you collect qualitative data? Develop an initial understanding of an issue or topic (or without bias) Explore a range of ideas and feelings about something Understand different perspectives Undercover underlying motivations Provide information needed to design a quantitative study Explain findings ngs from a quantitative study Closed-ended questions won t work well 2. You want to go deeper! Mixed Methods #txt Quantifying text: 90% of people interviewed shared that language was a barrier when asked What are barriers to accessing services? Combining data sources for analysis: 75% of people agreed AFTER training that adults can keep children safe from abuse. Those who agreed shared that the ability to learn from others was the most important factor in altering their views. Mixed Methods Analysis: Combine qualitative and quantitative methods of analysis (numbers & text)

5 When should you use mixed methods? Bring together results from many types of data collection Clarify or expand on results from one method to another Use results from one area to help inform another Seek discovery when data doesn t agree Expand the range of data to answer a question

6 Qualitative Analysis Evaluation Context This was the best experience of my entire life! One or two new ideas Qualitative Analysis Research Context

7 What are realistic ways to analyze qualitative data for useful program evaluation? Objectives Learn and practice three ways to interpret and evaluate qualitative analysis including: (1) categorizing qualitative responses into binary outcomes (2) categorizing qualitative responses into a multi-option coding system (3) a general group coding and interpretation process Participants will have the opportunity to explore questions and challenges related to qualitative data analysis and reporting options

8 Why might this question be limiting? Quantitative # When is it okay to have sex with someone who is drunk (ie- incapacitated)? A. Never B. Sometimes C. Always Qualitative txt When is it okay to have sex with someone who is drunk (ie- incapacitated)? Write answer here (1) categorizing qualitative responses into binary outcomes The analysis plan binary coding Code answers into two categories (ie- binary outcomes) Appropriate answer we will give this response a 1 Inappropriate answer give these responses a 0 (1) categorizing qualitative responses into binary outcomes Practice Binary Example

9 When is it okay to have sex with someone who is drunk (ie- incapacitated)? Appropriate = 1 Inappropriate= Response Appropriate? Code Never. If you are married. If you ve done it before with the same partner There are no circumstances when this is okay You shouldn t be drinking underage Complete the rest of your handout pre and post, add up the totals PRE Test Umm none If they are somewhere safe I don t think it is ever okay No, because that s taking advantage of them Sex is only for marriage When you use protection I don t know If they can still make decisions Never? It s not okay because you might get an STD (1) categorizing qualitative responses into binary outcomes Appropriate? Code

10 POST -Test Never Not okay ever None, because you re not sure if they want to You can t no matter what It s never okay I still think that if you talk about it before it s ok No because it s illegal No because you might not know the person Never!!! None (1) categorizing qualitative responses into binary outcomes Appropriate? Code Scores PRE-TEST POST-TEST

11 What is your rule? What is required for a good answer? It is super important to have this! You ll also want to remember that 1 is code for appropriate (and 0 = inappropriate) These items together can help to create a codebook! Question Coding Rule When is it okay to have sex with someone who is drunk? (ie- incapacitated) 1 = Appropriate answer O = Inappropriate answer (1) categorizing qualitative responses into binary outcomes To get a 1 participant has to identify that there is never any circumstance when this it is okay to have sex with someone who is drunk See the differences RULE: NO / NEVER / NONE / NOT OKAY (EXPLANATION DOESN T MATTER) RULE: NO / NEVER / NONE / NOT OKAY & EXPLANATION IS DUE TO IMMORAL, UNETHICAL, OR ILLEGAL BEHAVIOR Appropriate answers: PRE: 50% POST: 90% Appropriate answers: PRE: 30% POST: 80% Percent change: 80% Percent change: 167%

12 What is your rule? What is required for a good answer? Write it in the last space below on your sheet! In the future determine the rule before coding. (1) categorizing qualitative responses into binary outcomes Reporting option Before the training, % of participants could appropriately identify that consent cannot be given when a partner is drunk (ie incapacitated).* After the training, % of participants could appropriately identify that consent cannot be given when a partner is drunk (ie incapacitated).* This represented a percent change. *Using their own words, participants wrote their answers to the question, When is it okay to have sex with someone who is drunk? (ie- incapacitated) at pre and post training. These answers were coded into a binary system of appropriate and inappropriate responses. Please contact the program for more details on coding and analysis. (1) categorizing qualitative responses into binary outcomes

13 Key points Categorizing into two groups The good or correct answer will always be 1, the bad will always be 0 Remember that any answer that is incorrect will get a zero Write down the coding and requirements Once you have your requirements, stick to them! Remember that every analysis has error but if you are clear about how you code you can explain how you got your results. Keep results clear explain processes in a different place. (1) categorizing qualitative responses into binary outcomes Objectives Learn and practice three ways to interpret and evaluate qualitative analysis including: (1) categorizing qualitative responses into binary outcomes (2) categorizing qualitative responses into a multi-option coding system (3) a general group coding and interpretation process Participants will have the opportunity to explore questions and challenges related to qualitative data analysis and reporting options

14 Very similar to the process we just did, but now we can put responses into more than two categories! Gives us more detail! (2) categorizing qualitative responses into a multi-option coding system A = Never okay, w strong explanation B = Never okay, w no explanation Code: C= Never okay, but incorrect explanation D= Sometimes okay, with or without reason F= Always okay, with or without reason When is it okay to have sex with someone who is drunk (ie- incapacitated)? Response Never, because you can t be sure they want to Never. No because you might not know the person If you ve done it before with the same partner Anytime you want Grade A B C D F (2) categorizing qualitative responses into a multi-option coding system

15 New Example Bystander Intervention Stages of Change / Trans-theoretical Model Find this chart on the top of your handout: Practice Multi-Option STAGE DEFINITION* Pre-Contemplation Person does not notice a problem Contemplation Person aware of a problem, but does not want to take action Preparation Person would like to take action but does not have a plan or ability to act Action Person has a clear plan and feels confident to address the problem Maintenance Person offers that this sort of action is something they routinely do *Definitions are adapted from the stages of change model, but may not fit exact definitions from other sources

16 Situation Imagine the following situation where Ana, Julia, and Manny are your friends: Ana says, Your boyfriend hasn t kissed you yet, Julia? You re his girlfriend! I told you he was gay. Ana and Manny laugh. Julia does not laugh. Analysis questions for the stages of change -- Question 1: How do you feel about this situation? Question 2: What would you do if you were there when this happened? (2) categorizing qualitative responses into a multi-option coding system Response #1 Question 1: How do you feel about this situation? I feel like this sort of thing happens all the time it s not okay. Question 2: What might you do if these were your friends and you were with them when this happened? I m not sure. I would want to say something but I m afraid I might make Julia feel worse. What stage would you use for this response? Pre-Contemplation Contemplation Preparation Action Maintenance (2) categorizing qualitative responses into a multi-option coding system

17 Identify the stage for the five examples on your handout!!! STAGE DEFINITION* Pre-Contemplation Person does not notice a problem Contemplation Person aware of a problem, but does not want to take action Preparation Person would like to take action but does not have a plan or ability to act Action Person has a clear plan and feels confident to address the problem Maintenance Person offers that this sort of action is something they routinely do *Definitions are adapted from the stages of change model, but may not fit exact definitions from other sources Response Stage 2 Q1: Julia should probably be worried that he s gay, although it depends how long they ve been together. Q2: I don t know probably ask more questions to help Julia find out what s going on. (2) categorizing qualitative responses into a multi-option coding system

18 Response Stage 3 Q1: I don t know how I feel about the comment, but I would be worried that Julia wasn t laughing. Q2: I would definitely talk to Julia after to see if she s okay. (2) categorizing qualitative responses into a multi-option coding system Response Stage 4 Q1: I hate it when people assume that someone is gay just because they re waiting to have sex or kiss or something like that. Q2: I would tell Ana that s not cool. (2) categorizing qualitative responses into a multi-option coding system

19 Response Stage 5 Q1: Ana probably shouldn t be talking like that. Q2: Even though it s not nice what she said, I would stay out of it when you start getting involved in your friend s business it gets you into trouble. (2) categorizing qualitative responses into a multi-option coding system Using Results (fake data for 20 participants pre and post bystander training) Stages of Change for Bystander Intervention (darker = positive) 80% of participants moved along the stages of change. PRE-test POST-test PreContemplation Contemplation Preparation Action Maintenance (1) categorizing qualitative responses into binary outcomes Before training, only 20% of participants would intervene. After training, 60% of participants would intervene. (action/maintenance)

20 Key points Categorizing multiple options Create clear definitions of categories. Stick to definitions even when it s tough. You get to be the decider! If you need to change definitions that s okay. Go back and make sure that you code all responses with the same rules. Try not to overthink the response this will take forever and the goal is to be able to go through categories relatively quickly. Don t be afraid to throw data out or put it in a NO CODE category. Be creative with reporting (2) categorizing qualitative responses into a multi-option coding system Objectives Learn and practice three ways to interpret and evaluate qualitative analysis including: (1) categorizing qualitative responses into binary outcomes (2) categorizing qualitative responses into a multi-option coding system (3) a general group coding and interpretation process Participants will have the opportunity to explore questions and challenges related to qualitative data analysis and reporting options

21 General Group Coding Method A Sort Then Name -Goal: Walk away with themes or categories inclusive of all voices & responses (3) A general group coding and interpretation process General Group Coding What are strengths, or protective factors, in the Latina community? Results from a Midwestern focus group with Latin@ advocates. (3) A general group coding and interpretation process

22 What are strengths, or protective factors, in the Latina community? Family connection Fight for what you want Communication Help to each other (in some circumstances) Spirituality as a tool in the healing process Adaptation / Acculturation showcasing ability to adapt to different environments / lifestyles Caballerism Chivalry, the way the ideal man should behave Familismo/comunidad Family/Community Intersections Family oriented Compassionate Determined Religion Traditions Educated Hardworking Supportive Family Happy Patience Love (3) A general group coding and interpretation process Focus group results (in list form) Analysis Plan!!!! Method A: Sort then Name 1. Put the items in groups that belong together. 2. Give each box a title. Note: Can use power point, hard copy paper, excel, word, or other programs. What are strengths, or protective factors, in the Latina community? Family connection Fight for what you want Communication Help to each other (in some circumstances) Spirituality as a tool in the healing process Adaptation / Acculturation showcasing ability to adapt to different environments / lifestyles Caballerism Chivalry, the way the ideal man should behave Familismo/comunidad Family/Community Intersections Family oriented Compassionate Determined Religion Traditions Educated Hardworking Supportive Family Happy Patience Love (3) A general group coding and interpretation process

23 Reporting results... Through a qualitative analysis process of sorting and naming, we included all data and determined that the primary strengths, or protective factors of the Latina community are: 1. Family 2. Title 2 3. Title 3 4. Title 4 5. Title (3) A general group coding and interpretation process Method A (Sort than name) is good for -- What about focus groups? Good method when each respondent will be answering a question independently. Good for overview of process (less depth).

24 Focus Group Data Sort of different than other data not looking to necessarily quantify or find all themes, but to explore deeper and find shared experiences. Method B: Name first then use method A (sort and re-name) (Note - Open focus group transcript start with family data) How does family act as a protective factor? (3) A general group coding and interpretation process First name each section with a short phrase Compile all names together and sort into themes!

25 How does family act as a strength or protective factor? Good & bad Good when supportive Bad when family harms each other Preserve family unit, leads to growth Create professional families Create your own families as you move Use family model for academia Create support Good & bad Judgement with divorce Family stays together Other families are not as connected Other families pretend to have no issues Communication with family is a strength No family communication with prevention Sex message = don t do it Family didn t bring up unhealthy relationship Good communication and yet missing Lack parental guidance Lack of parents redirecting Theme 1 Theme 3 Theme 5 Theme 2 Theme 4 Theme 6 (3) A general group coding and interpretation process How does family act as a strength or protective factor? Good Supportive, Stay Together Good when supportive Preserve family unit, leads to growth Create family units Communication Good & bad Family stays together Other families are not as connected Create professional families Create your own families as you move Use family model for academia Create support Communication with family is a strength Good communication and yet missing Other families pretend to have no issues Good & bad Bad Harms Each Other Bad when family harms each other Divorce is judged Judgement with divorce Lack of parental direction Family didn t bring up unhealthy relationship Lack parental guidance Lack of parents redirecting Lack of communication for prevention No family communication with prevention Sex message = don t do it (3) A general group coding and interpretation process

26 Practice Use 3-page handout, What does it mean to be Latina in the United States? Alone Name each section with a word or short phrase 2. Put each word or short phrase on a separate note card Together Sort the post it notes into groups that are similar 4. Name the similar groups. 5. Write down final themes and results. (3) A general group coding and interpretation process Difficult 3 rd class citizens Don t fit in Difficult 3 rd class citizens Don t fit in 1 Difficult 3 rd class citizens Don t fit in It is hard 2 Feel de-valued Difficult 3 rd class citizens Don t fit in Results: It is hard, feel de-valued, etc. What does it mean to be Latina in the United States? 1. Difficult question 2. Feel misunderstood (both adults & children) 3. It s exhausting to have to explain to others & to children about origins, differences, language, geography, immigration status, immigration story 4. Feels like being a minority, with stigma 5. Have to work really hard 6. It s double the efforts to meet needs, without recognition 7. Politically devalued 8. Tokenized 9. Have to deal with constant assumptions (3) A general group coding and interpretation process

27 Key points General coding process It is easier to work with written data takes more time to transcribe oral information or focus groups With interviews or open-ended survey questions, sort responses immediately and then name the groups With focus groups (or for deeper explanations) name the responses first, then sort into groups Most of these processes can be done with pairs or small groups. This can take time but will lead to deeper findings! (3) A general group coding and interpretation process Objectives Learn and practice three ways to interpret and evaluate qualitative analysis including: (1) categorizing qualitative responses into binary outcomes (2) categorizing qualitative responses into a multi-option coding system (3) a general group coding and interpretation process Participants will have the opportunity to explore questions and challenges related to qualitative data analysis and reporting options

28 Resources American Evaluation Association Qualitative Methods Topical Interest Group Resources Visualizing Qualitative Data (Eval11 Session 744) (if link is broken put exact text into search engine) 92af-d181aaf2b7db Percent change calculator Focus Group and Interview General Resources Groups.php WCASA Evaluation Resources Thank you!! ROSE HENNESSY, MPH PREVENTION & EVALUATION COORDINATOR WISCONSIN COALITION AGAINST SEXUAL ASSAULT