Working with data: Data analyses April 8, 2014
Housekeeping notes This webinar will be recorded, and will be available on the Centre s website as an educational resource The slides have been sent to participants Log in with a phone whenever possible for optimal audio quality We have staff online to assist with any technical difficulties There will be a short evaluation survey for all webinar attendees at the completion of the webinar.
Housekeeping notes This webinar will be followed by a question and answer period, however questions are encouraged throughout the presentation. Questions can be submitted electronically or verbally. Specifics around this process will be clarified at the end of the webinar.
Welcome and introductions
Today s presenters Katherine Thompson Program Associate, Support Services Ontario Centre of Excellence for Child and Youth Mental Health Chris Conley Research Analyst Accountability and Assessment Durham District School Board Executive Lead Barrie Region MISA
About the Centre We bring people and knowledge together to strengthen the quality and effectiveness of mental health services for children, youth and their families and caregivers. Three strategic goals: Learning Collaboration Leadership Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes. Build and develop collaborative partnerships to sustain capacity within mental health services. Be a true learning organization and lead by example.
Three part series: Data Collection Data Management Data Analysis
How about you Are you involved in data analysis? I analyze data I am a decision maker/program supervisor/clinician interested in the results I am involved in collecting data I am not involved in data analysis Other
Goals of this webinar To share information on: Data analysis planning Introductory quantitative and qualitative data analyses topics Reporting on data analysis
Before you analyze the data
Decisions.decisions
Key steps to data analyses planning 1. Develop evaluation questions 2. Decide what data will be analyzed, how and when 3. Decide who is responsible for conducting the analysis 4. Choose method to analyze data (descriptive statistics, inferential statistics, content analysis) 5. Pilot test measurement tools and analysis 6. Consider reporting mechanisms
Evaluation analysis plan Process Evaluation Question Measures Plan for Analysis Outcome Evaluation Question Measures Plan for Analysis
Considerations in analysis planning Timelines Resources Skill level Program needs Linking the right questions with the right analysis
Planning tools Evaluation Question Data collection method Data to be analyzed Type of analysis Needed resources Person(s) responsible Timelines
Analyzing
Key steps involved in data analyses 1. Organize data 2. Analyze data 3. Interpret data analysis results 4. Identify limitations (Taylor-Powell, 2008)
Organizing the data Data coding Check and address missing data Calculate response rate Data code book to manage and monitor data (See Working with Data: Part 2 Data Management)
Making sense of the numbers 10
Sources of evaluation data Quantitative Qualitative Mixed methods surveys questionnaires tests existing databases interviews focus groups observations document review combines both qualitative and quantitative
What are your evaluation questions? If you are trying to learn... If you are trying to learn... How many? How much? What percentage? How often? What is the average amount? What worked best? What did not work well...? What do the numbers mean? How was the project useful...? What factors influenced success or failure? Use a quantitative method Use a qualitative method Onley & Barnes, 2008
Qualitative or quantitative? Qualitative Quantitative Goal Complete description Classification and counting to explain phenomenon Approach Inductive Deductive Focus Role of evaluator Subjective (individual s interpretations) Evaluator is the instrument Objective (precise measurement) Evaluator uses instruments Nature of data Words, pictures, objects Numbers and statistics Generalizability Low High
Quantitative data analyses Summarize data using descriptive statistics o Example: Determine frequency, percentages, means Organize data into tables or charts to help answer evaluation questions If assessing outcomes, compare findings across participants and program characteristics If assessing change, compare findings across time and/or conditions Write a brief description of your results Adapted from Olney & Barnes, 2008
Answering the evaluation questions in a way that is accurate, credible, and useful is key to a successful evaluation. Can you share how you make those decisions in selecting the right statistical analysis that will contribute to producing results that will be accurate, credible, relevant and ultimately answer these questions? Answer: Chris Conley
Analysis Tools Data Action People Question
Quantitative statistical procedures summary Numerical data Usual procedure Test of significance Change over time 1 group Mean N/A Paired t-test 2 groups Comparison of means Categorical data 1 group Frequency distribution Cross-tabulation t-test N/A Chi-square Cross-tabulation (McNemar test) (American Academy of Pediatrics, 2008)
Quantitative data analysis Descriptive Statistics Summarize characteristics Describe/explore characteristics Compare characteristics Examples Frequency, percentages Central tendency (mean, mode, median) Variance, standard deviation Inferential Statistics Make generalizations about population based on sample Assess effects of intervention/tx on outcomes measures Compare and assess for statistical significance Examples Chi square t-test Analysis of Variance (ANOVA) Pearson correlations Regression
Consider Sample o o Size Participant characteristics Type of data o o o Nominal Ordinal Ratio and interval Distribution o o Normal Non-normal
Interpretation-make sense of the information Look for: o o o Patterns High/low numbers Expected/unexpected results Meaning of: o o o o Numbers Words Comments Observations Highlight key points and lessons learned Taylor-Powell, E., Steele, S., & Douglah, M. (1996).
Data analysis is an activity that requires time and some degree of skill depending on the complexity of the data set. How do you ensure that data that are collected do not remain in a filing cabinet or database? How does it support the analysis of program evaluation data? Answer: Chris Conley
I II III IV x y x y x y x y 10.0 8.04 10.0 9.14 10.0 7.46 8.0 6.58 8.0 6.95 8.0 8.14 8.0 6.77 8.0 5.76 13.0 7.58 13.0 8.74 13.0 12.74 8.0 7.71 9.0 8.81 9.0 8.77 9.0 7.11 8.0 8.84 11.0 8.33 11.0 9.26 11.0 7.81 8.0 8.47 14.0 9.96 14.0 8.10 14.0 8.84 8.0 7.04 6.0 7.24 6.0 6.13 6.0 6.08 8.0 5.25 4.0 4.26 4.0 3.10 4.0 5.39 19.0 12.50 12.0 10.84 12.0 9.13 12.0 8.15 8.0 5.56 7.0 4.82 7.0 7.26 7.0 6.42 8.0 7.91 5.0 5.68 5.0 4.74 5.0 5.73 8.0 6.89
Ingredients for successful data analysis Reaching the right people Active engagement Timely return of information Making informed decisions to find useful, credible and targeted answers to evaluation questions
Strengths of quantitative approach Can measure depth and breadth of an implementation (e.g., the number of people who participated, the number of people who completed the program). Collected before and after an intervention can show its outcomes and impact. Generalizability (if the sample represents the population) Ease of analysis Consistency and precision (if collected reliably) More easily replicated
Identify limitations of quantitative approach See explanations for any external and internal factors that may have affected the results Identify any limitations such as useful information that was not collected Bias in respondents answers Low return rates How might affect results (reflection) What you may be able to do about them in the future Taylor-Powell, E., Steele, S., & Douglah, M. (1996).
Key points to remember: Identify in advance resources needed to implement data analysis process Ensure all data analysis team members are trained and understand process Include a strategy for monitoring and quality control Involve key stakeholders with updates and opportunity for feedback
It is important to share evaluation findings through a range of mechanisms depending on the stakeholder group. What have you found to be some key ingredients in effectively mobilizing that knowledge across stakeholder groups? Answer: Chris Conley
Data Visualization Purpose Explore Explain Stakeholder Expectations Explain Explore Stakeholder responses: Data is too aggregated. Data is oversimplified. Visualization too limited Stakeholder responses: Too complicated. Too technical. Doesn t make sense
Telling a story. increased aware the hope awesome happy challenge skill improved access decreased declined success knowledge
A qualitative approach can Help to tell the program s story by incorporating participants voices Provide a deep, more complete understanding of the program and how it works (particularly complex programs), which can be helpful in decision-making Offer insight into why things may or may not be working well
Qualitative analyses process Read through transcripts and notes to develop code list Write brief description of each theme Code all interview data systematically Organize the coded text by code or theme Interpret findings
Interpretation Search for common themes (codes/categories) across interviews (content analysis), question by question Check for confirmatory and non-confirmatory cases across interviews Group common items together, and identify relationships between different categories
Strengths of qualitative research Answers the why question Examines personal meaning Examines complex group behaviors Provides context Allows the study of dynamic processes Helps determine questions and types of follow-up research Contributes to theory development Useful for describing complex phenomena Facilitates knowledge exchange
Limitations of qualitative research Interpretations are easily influenced by personal experiences and biases Lack of generalizability Cannot test hypotheses Time intensive Requires skills and considerable training
Mixed methods A research design that combines quantitative and qualitative approaches and methods in order to improve understanding of a problem Benefits o o o o Compensates for weaknesses of each approach More comprehensive than one method alone Accommodates questions that neither approach could answer Increases potential for collaboration
Computer aided analysis Quantitative analysis SPSS Excel Access R Qualitative analysis NVivo Atlas-ti NUD*IST
Excel and SPSS are two of the more widely used computer programs for data analysis. Excel is a spreadsheet program while SPSS a statistical analysis program. Can you speak to the similarities and differences between the two programs and how you decide which one to use in your work? Answer: Chris Conley
Excel SPSS Entering Data Cleaning Data Analyzing Data Data Visualization Complex Tables Interactive
After you analyze the data.
After you have analyzed the data Report the information Mobilize knowledge Act on findings
You have identified that both Excel and SPSS can be used for data visualization, however you prefer using Excel for data visualization. Can you show us some examples of how t o maximize our use of Excel to display our results most effectively and maximize understandability? Answer: Chris Conley
With Special Needs Without Special Needs Without Special Needs 82 Academic or University Academic or University 74 Living with Both Parents Male 68 64 70 Grades 11-12 Marks (Mean) 66 Parents with Post-Secondary Confirmed University Acceptance 60 58 58 Median Family Income 55 Confirmed University Acceptance First Generation 49 Second Generation 45 White 44 49 47 First Generation Parents with Post-Secondary East Asian Grades 11-12 Marks (Mean) Living with Both Parents South Asian 32 Third Generation 31 36 32 With Special Needs Median Family Income / Social Engagement Academic Engagement Confirmed College Acceptance 24 23 Black 19 15 13 23 East Asian 20 South Asian 17 14 Black / Second Generation / Confirmed College Acceptance White Third Generation Male Academic Engagement 8 6 4 9 4 Social Engagement
Collaborative Data Visualization
Provincial Exploration Through Visualization
http://dugroup.ca
Remember... Don t complicate things Keep it simple
Additional resources Webinars Reliable Change Quantitative analyses Qualitative evaluation Overview: Developing Excel database Mini-toolkits Modules
Questions?
Questions or comments? To submit questions electronically, use the question box located in your control panel To submit questions verbally, use the raised hand icon also located in the control panel
Contact information For more information on this webinar or topic, please contact: Katherine Thompson Program Associate, Support Services Ontario Centre of Excellence for Child and Youth Mental Health kathompson@cheo.on.ca
Contact us centre@cheo.on.ca 613-737-2297 x. 3316 www.excellenceforchildandyouth.ca @CYMH_ON
Evaluation Please complete the survey at the end of this webinar. Your feedback is very important to us, so we thank you for taking the time to share your thoughts!