Working with data: Data analyses April 8, 2014

Transcription

1 Working with data: Data analyses April 8, 2014

2 Housekeeping notes This webinar will be recorded, and will be available on the Centre s website as an educational resource The slides have been sent to participants Log in with a phone whenever possible for optimal audio quality We have staff online to assist with any technical difficulties There will be a short evaluation survey for all webinar attendees at the completion of the webinar.

3 Housekeeping notes This webinar will be followed by a question and answer period, however questions are encouraged throughout the presentation. Questions can be submitted electronically or verbally. Specifics around this process will be clarified at the end of the webinar.

4 Welcome and introductions

5 Today s presenters Katherine Thompson Program Associate, Support Services Ontario Centre of Excellence for Child and Youth Mental Health Chris Conley Research Analyst Accountability and Assessment Durham District School Board Executive Lead Barrie Region MISA

6 About the Centre We bring people and knowledge together to strengthen the quality and effectiveness of mental health services for children, youth and their families and caregivers. Three strategic goals: Learning Collaboration Leadership Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes. Build and develop collaborative partnerships to sustain capacity within mental health services. Be a true learning organization and lead by example.

7 Three part series: Data Collection Data Management Data Analysis

8 How about you Are you involved in data analysis? I analyze data I am a decision maker/program supervisor/clinician interested in the results I am involved in collecting data I am not involved in data analysis Other

9 Goals of this webinar To share information on: Data analysis planning Introductory quantitative and qualitative data analyses topics Reporting on data analysis

10 Before you analyze the data

11 Decisions.decisions

12 Key steps to data analyses planning 1. Develop evaluation questions 2. Decide what data will be analyzed, how and when 3. Decide who is responsible for conducting the analysis 4. Choose method to analyze data (descriptive statistics, inferential statistics, content analysis) 5. Pilot test measurement tools and analysis 6. Consider reporting mechanisms

13 Evaluation analysis plan Process Evaluation Question Measures Plan for Analysis Outcome Evaluation Question Measures Plan for Analysis

14 Considerations in analysis planning Timelines Resources Skill level Program needs Linking the right questions with the right analysis

15 Planning tools Evaluation Question Data collection method Data to be analyzed Type of analysis Needed resources Person(s) responsible Timelines

16 Analyzing

17 Key steps involved in data analyses 1. Organize data 2. Analyze data 3. Interpret data analysis results 4. Identify limitations (Taylor-Powell, 2008)

18 Organizing the data Data coding Check and address missing data Calculate response rate Data code book to manage and monitor data (See Working with Data: Part 2 Data Management)

19 Making sense of the numbers 10

20 Sources of evaluation data Quantitative Qualitative Mixed methods surveys questionnaires tests existing databases interviews focus groups observations document review combines both qualitative and quantitative

21 What are your evaluation questions? If you are trying to learn... If you are trying to learn... How many? How much? What percentage? How often? What is the average amount? What worked best? What did not work well...? What do the numbers mean? How was the project useful...? What factors influenced success or failure? Use a quantitative method Use a qualitative method Onley & Barnes, 2008

22 Qualitative or quantitative? Qualitative Quantitative Goal Complete description Classification and counting to explain phenomenon Approach Inductive Deductive Focus Role of evaluator Subjective (individual s interpretations) Evaluator is the instrument Objective (precise measurement) Evaluator uses instruments Nature of data Words, pictures, objects Numbers and statistics Generalizability Low High

23 Quantitative data analyses Summarize data using descriptive statistics o Example: Determine frequency, percentages, means Organize data into tables or charts to help answer evaluation questions If assessing outcomes, compare findings across participants and program characteristics If assessing change, compare findings across time and/or conditions Write a brief description of your results Adapted from Olney & Barnes, 2008

24 Answering the evaluation questions in a way that is accurate, credible, and useful is key to a successful evaluation. Can you share how you make those decisions in selecting the right statistical analysis that will contribute to producing results that will be accurate, credible, relevant and ultimately answer these questions? Answer: Chris Conley

25 Analysis Tools Data Action People Question

26

27

28 Quantitative statistical procedures summary Numerical data Usual procedure Test of significance Change over time 1 group Mean N/A Paired t-test 2 groups Comparison of means Categorical data 1 group Frequency distribution Cross-tabulation t-test N/A Chi-square Cross-tabulation (McNemar test) (American Academy of Pediatrics, 2008)

29 Quantitative data analysis Descriptive Statistics Summarize characteristics Describe/explore characteristics Compare characteristics Examples Frequency, percentages Central tendency (mean, mode, median) Variance, standard deviation Inferential Statistics Make generalizations about population based on sample Assess effects of intervention/tx on outcomes measures Compare and assess for statistical significance Examples Chi square t-test Analysis of Variance (ANOVA) Pearson correlations Regression

30 Consider Sample o o Size Participant characteristics Type of data o o o Nominal Ordinal Ratio and interval Distribution o o Normal Non-normal

31 Interpretation-make sense of the information Look for: o o o Patterns High/low numbers Expected/unexpected results Meaning of: o o o o Numbers Words Comments Observations Highlight key points and lessons learned Taylor-Powell, E., Steele, S., & Douglah, M. (1996).

32 Data analysis is an activity that requires time and some degree of skill depending on the complexity of the data set. How do you ensure that data that are collected do not remain in a filing cabinet or database? How does it support the analysis of program evaluation data? Answer: Chris Conley

33

34 I II III IV x y x y x y x y

35

36

37 Ingredients for successful data analysis Reaching the right people Active engagement Timely return of information Making informed decisions to find useful, credible and targeted answers to evaluation questions

38 Strengths of quantitative approach Can measure depth and breadth of an implementation (e.g., the number of people who participated, the number of people who completed the program). Collected before and after an intervention can show its outcomes and impact. Generalizability (if the sample represents the population) Ease of analysis Consistency and precision (if collected reliably) More easily replicated

39 Identify limitations of quantitative approach See explanations for any external and internal factors that may have affected the results Identify any limitations such as useful information that was not collected Bias in respondents answers Low return rates How might affect results (reflection) What you may be able to do about them in the future Taylor-Powell, E., Steele, S., & Douglah, M. (1996).

40 Key points to remember: Identify in advance resources needed to implement data analysis process Ensure all data analysis team members are trained and understand process Include a strategy for monitoring and quality control Involve key stakeholders with updates and opportunity for feedback

41 It is important to share evaluation findings through a range of mechanisms depending on the stakeholder group. What have you found to be some key ingredients in effectively mobilizing that knowledge across stakeholder groups? Answer: Chris Conley

42 Data Visualization Purpose Explore Explain Stakeholder Expectations Explain Explore Stakeholder responses: Data is too aggregated. Data is oversimplified. Visualization too limited Stakeholder responses: Too complicated. Too technical. Doesn t make sense

43

44

45

46

47

48

49

50 Telling a story. increased aware the hope awesome happy challenge skill improved access decreased declined success knowledge

51 A qualitative approach can Help to tell the program s story by incorporating participants voices Provide a deep, more complete understanding of the program and how it works (particularly complex programs), which can be helpful in decision-making Offer insight into why things may or may not be working well

52 Qualitative analyses process Read through transcripts and notes to develop code list Write brief description of each theme Code all interview data systematically Organize the coded text by code or theme Interpret findings

53 Interpretation Search for common themes (codes/categories) across interviews (content analysis), question by question Check for confirmatory and non-confirmatory cases across interviews Group common items together, and identify relationships between different categories

54 Strengths of qualitative research Answers the why question Examines personal meaning Examines complex group behaviors Provides context Allows the study of dynamic processes Helps determine questions and types of follow-up research Contributes to theory development Useful for describing complex phenomena Facilitates knowledge exchange

55 Limitations of qualitative research Interpretations are easily influenced by personal experiences and biases Lack of generalizability Cannot test hypotheses Time intensive Requires skills and considerable training

56 Mixed methods A research design that combines quantitative and qualitative approaches and methods in order to improve understanding of a problem Benefits o o o o Compensates for weaknesses of each approach More comprehensive than one method alone Accommodates questions that neither approach could answer Increases potential for collaboration

57 Computer aided analysis Quantitative analysis SPSS Excel Access R Qualitative analysis NVivo Atlas-ti NUD*IST

58 Excel and SPSS are two of the more widely used computer programs for data analysis. Excel is a spreadsheet program while SPSS a statistical analysis program. Can you speak to the similarities and differences between the two programs and how you decide which one to use in your work? Answer: Chris Conley

59 Excel SPSS Entering Data Cleaning Data Analyzing Data Data Visualization Complex Tables Interactive

60 After you analyze the data.

61 After you have analyzed the data Report the information Mobilize knowledge Act on findings

62 You have identified that both Excel and SPSS can be used for data visualization, however you prefer using Excel for data visualization. Can you show us some examples of how t o maximize our use of Excel to display our results most effectively and maximize understandability? Answer: Chris Conley

63 With Special Needs Without Special Needs Without Special Needs 82 Academic or University Academic or University 74 Living with Both Parents Male Grades Marks (Mean) 66 Parents with Post-Secondary Confirmed University Acceptance Median Family Income 55 Confirmed University Acceptance First Generation 49 Second Generation 45 White First Generation Parents with Post-Secondary East Asian Grades Marks (Mean) Living with Both Parents South Asian 32 Third Generation With Special Needs Median Family Income / Social Engagement Academic Engagement Confirmed College Acceptance Black East Asian 20 South Asian Black / Second Generation / Confirmed College Acceptance White Third Generation Male Academic Engagement Social Engagement

64

65

66

67

68

69

70

71

72

73 Collaborative Data Visualization

74 Provincial Exploration Through Visualization

75

76 Remember... Don t complicate things Keep it simple

77 Additional resources Webinars Reliable Change Quantitative analyses Qualitative evaluation Overview: Developing Excel database Mini-toolkits Modules

78 Questions?

79 Questions or comments? To submit questions electronically, use the question box located in your control panel To submit questions verbally, use the raised hand icon also located in the control panel

80 Contact information For more information on this webinar or topic, please contact: Katherine Thompson Program Associate, Support Services Ontario Centre of Excellence for Child and Youth Mental Health

81 Contact us x

82 Evaluation Please complete the survey at the end of this webinar. Your feedback is very important to us, so we thank you for taking the time to share your thoughts!