Main Objectives. Quantitative Data Analysis (Advanced) Steps Involving Data. Types of Data. HOPE: Summer Institute June 14 18, 2010, Winnipeg Manitoba

Transcription

1 HOPE: Summer Institute June 8, 200, Winnipeg Manitoba Quantitative Data Analysis (Advanced) June, 20 Presenter: Aynslie Hinds Main Objectives To learn about basic inferential statistics and when to perform the different types of analyses To understand how to interpret output of some basic analyses To consider the value and limitations of various quantitative methods Steps Involving Data Types of Data Design and test data collection instruments Collect the data Data entry Clean the data Analyze the data Interpret the results Transformation

2 Scales of Measurement Scale Measurement Scale Example Nominal Ordinal Interval Do students study? categories = yes and no Who studies more? Scale adds more than Who studies more? Students rate their studying on a point scale Melissa yes Jeff Yes Gina No Melissa > Jeff> Gina Melissa (always) Jeff 3 (sometimes) Gina (never) Scale of Measurement Understanding the type of data is key to knowing How to create a data file The correct method for the analyzing data and presenting the results Ratio Who studies more? Measure the amount time spent studying Melissa 2 hours/day Jeff 2 hours/day Gina 0 hours/day Statistics Statistical investigation and analyses of data fall into two broad categories: Descriptive statistics Inferential statistics Descriptive Statistics Methods for summarizing and presenting data May involve: presentation of data in graphical or tabular form calculation of summary statistical measures E.g., bar charts, histograms, scatterplots, averages, variance

3 Typical Value Measures of Central Tendency Mean Median Mode Describes only one important aspect of the distribution of the data Need to consider the amount of variation or scatter Example Data Set Data Set Mean = Median = Mode = 0. Range 2. Variance 3. Standard deviation Measures of Dispersion

4 Normal Distribution Widely observed in natural and behavioral sciences Description: Most results are close to the mean (typical) Few results are atypical The more atypical a result, the less frequent it occurs Normal Distribution Many statistical tests are based on the assumption of normality Parametric VS Non Parametric Normal Curve Sampling variability with repeated sampling Number of cigarettes smoked yesterday Population of seniors students at one high school in Winnipeg Population Mean μ X = Samples 8 mean =.0 mean = mean =. mean =.3 mean =. Sample Statistics Mean: x Variance: s 2 Standard Deviation: s Proportion: p Estimates vs Parameters Population Parameters Mean: µ Variance: σ 2 Standard Deviation: σ Proportion: ρ We know or observe x from a sample, but we don t know or observe µ Can observe a sample statistic and use it as an estimate of the true, unknown population parameter

5 Inferential Statistics Inferential Analysis Involves: Using sample information to draw inferences or test hypotheses about a characteristic of a population Making inductive generalizations from the particular (the sample) to the general (the population) Hypothesis Testing & Estimation Object of Study: Population Sample Explain Parameters (true population mean, true population proportion) Infer Statistics (sample mean, standard deviation, proportion, etc) Compute Estimation: Asking and Answering Questions What is the true proportion of pregnant women who will quit smoking if they undergo a smoking cessation program? What is the true mean change in self esteem scores of individual participating in a skill based employment training program between pre and post program? What is the true mean change in perceptions of safety among community members pre and post program (e.g., improved street lightening, graffiti removal, etc.)? Estimation Process of calculating some statistic that is offered as an approximation (a guess ) to an unknown population parameter from which the sample was drawn Two methods for providing an estimate of a parameter Point estimate Interval estimation (i.e., confidence interval)

6 Interval Estimation/ Confidence Interval Range of values (interval) that is believed to contain the parameter of interest together with a certain degree of confidence (probabilistic statement) in the assertion that the interval does contain the parameter Confidence Level Describes the chance or probability that intervals of this kind capture the population value in the long run Levels of confidence: 0%, %, 8%, % Demonstration Each sample gives rise to a point estimate and an associated interval estimate of µ. True Mean Balance Precision (interval Width) Reliability (Confidence Level) Intervals don t contain µ Intervals contain µ

7 Hypothesis Testing: Asking and Answering Questions Can counseling can reduce smoking rates during pregnancy? Can a school based Just Say No campaign reduce drug use? Has participants self esteem increased as a result of participating in a skills based employment training program? Does having a safety outreach worker in the community increase community members sense of safety? Statistical Tests Lots of different statistical tests Challenge to know which one to use Parametric VS Non Parametric Decision Tree for Statistical Tests Regression: For each value of x, the corresponding values of y have a distribution that is bell-shaped. For different values of x, the distributions of the corresponding y-values all have the same Assumptions variance. For the different values of x, the distributions of the Numeric Variables corresponding y-values have means that lie along a straight line. y-values are independent. Relationship Between Single Sample One IV Two or more IVs 2 or More Variables (Factorial (IV Numeric) Difference Between Designs) ) Normally Groups Linear DIstributed 2) n 30 Correlation and Correlation: Regression Two or Random sample Two Groups Relationship is linear More Groups σ Known Pairs of data must have a bivariate normal distribution Independent ANOVA: Single Sample Related Groups Groups CRD Samples come from populations z test -with the same variance Sample size is large (n 30) -with a normal distribution ANOVA: OR population of paired differences is normally distributed Related RBD or σ Unknown Paired Difference Groups Repeated t test Measures Parametric Paired-difference t-test Independent t-test One-way ANOVA Equivalent Tests Non-Parametric Wilcoxon Signed Ranks test Wilcoxon Rank-Sum test Mann-Whitney U-test Kruskal-Wallis test Single Sample Independent t test Groups Normally Distributed 3 cases: )Population standard deviations (σ2) known 2)σ2 assumed equal Independent t test 3)σ2 not assumed equal IV: Unrelated 2 IV: Related Mixed Design Linear correlation Rank correlation Note: Numerical variable = Quantitative variable CRD: Completely Randomized Design RBD: Randomized Block Design

8 2 2 Decision Tree for Statistical Tests 2 Related Groups. Pre Post Study Before Intervention After Same People 2. Matched on Relevant Characteristics Expected Values Expected Values Ordinal Regression Female years ses: low IQ = 0 2 Match on socioeconomic background age sex 2 Test Statistic = Paired Difference t-test npˆ, npˆ, n pˆ ( ) and are all > n pˆ ( ) 2 2 Male 2 years ses: high IQ = 20 Is there a statistically significant difference in the means between the two conditions? Test Statistic = Independent t-test Classroom A 2 Unrelated Groups Is there a statistically significant difference in the means between the two conditions? Classroom B Two or More Groups/Conditions Analysis of Variance (ANOVA) Extension of an independent t test (CRD) Extension of a paired difference t test (RBD, repeated measures) One way ANOVA Two way ANOVA (Factorial Design) Follow Up Analyses Multiple Comparison Procedures Contrasts

9 Example Example Students Group Independent Variable Participate in exchange program Dependent Variable Measure attitudes toward immigrants Students Group Dependent Variable: Before Measure attitudes toward immigrants Independent Variable Participate in exchange program Dependent Variable: After Measure attitudes toward immigrants Students Group 2 Do not participate in exchange program Measure attitudes toward immigrants Students Group 2 Measure attitudes toward immigrants Do not participate in exchange program Measure attitudes toward immigrants Posttest-Only Non-Equivalent Control Group Design Analysis: Independent t-test Pretest-Posttest Non-Equivalent Control Group Design Factorial Design (Mixed) Example Pretest Treatment Posttest 8 Is the improvement due to the program or some other factors? Winnipeg Winnipeg plant Regina plant Average productivity for month prior to instituting flextime Average productivity for month prior to instituting flextime in Winnipeg Flextime instituted for months None Pretest-Posttest Non-Equivalent Control Group Design Factorial Design (CRD) Average productivity during th month of flextime Average productivity during th month that flextime is in effect in Winnipeg Productivity Pre Something other than flextime produced the improvement (e.g., history, maturation) because both plants increased productivity. Example explanations: National election/olympic victories/canadian hockey team wins championship between pre and post tests that workers everywhere felt more optimistic leading to increased productivity or improvement due to increased experience. Time Regina Post

10 2 Regina 0 Winnipeg Productivity 0 8 Winnipeg Productivity 8 Regina Pre Regina scores might reflect a ceiling effect (i.e., their productivity level is so high to begin with that no further improvement could be possible). Might see parallel lines if an increase was possible. Because Winnipeg started so low the increase might be a regression to the mean effect rather than a true one. Time Post Pre Strongest support for program effectiveness. Treatment group begins below control group, but surpasses the control group by the end. Regression can be ruled out as causing improvement because one would expect to raise the scores only to the Time level of the control group and not beyond it. Post Hawthorne Effect Contingency Table Cross tabulation Two way table Enumeration or count data classified according to two criteria Classes/categories from one criterion may be represented by the rows Classes/categories for the other criterion by the columns 2 x Contingency Table Completed the Program Sex Yes No Total Male Female Total A cell of the table is formed by the intersection of a row and column. Number inside a cell is called the joint frequency.

11 Chi Square Test of Independence Goal: To determine whether two attributes (categorical variables) are independent Correlation, r Are the two continuous variables measured on the same people related? Assess the strength and direction (of linear relationships) Example Is there a relationship between the number of sessions participants attended a nutrition program and their confidence rating in cooking healthy meals? Properties of the Correlation Coefficient. Positive r (r>0) indicates a positive linear or direct association As x increases, y increases (best fit line slopes up) 2. Negative r (r<0) indicated a negative linear or indirect association As x increases, y decreases (best fit line slopes down) 3. r always between and + ( r ) Values close to + or show strong linear associations (points are scattered closely around a line ) r = + or a perfect relationship (all the points fall on a line) Values near 0 show no/weak linear associations Regression Models Linear, logistic, ordinal, proportional hazards, etc. analysis depends on outcome variable Takes into account all sorts of explanations to explain an outcome; able to tease apart the individual contributions of the explanatory variables Which of the following variables best predict/explain level of confidence number of sessions attended, sex, age,

12 Logic of Hypothesis Testing Population I believe the population mean age is 0 (hypothesis). Random sample Mean X = 20 Reject Reject hypothesis! Not Not close. close. Sampling Distribution It is unlikely that we would get a sample mean of this value... Logic of Hypothesis Testing if in fact this were the population mean μ = 0 H 0 (Initial Assumption)... therefore, we reject the hypothesis that μ = 0. Sample Mean p value 0.0 or 0.0 p = probability Reject the null hypothesis (initial assumption) if p value = times out of 00 = % The probability of seeing this result (or one that s more extreme), by chance if the initial assumption is correct, is less than or equal to % p value > 0.0 p value = 0., p value = 0., etc. No evidence against the null hypothesis (initial assumption), therefore stick with it No statistically significant result

13 p value Produced by Statistical Program Statistical programs usually provide the p value for a two tailed/non directional alternative hypothesis NOTE: If the results are in the direction (+/ ) expected, divide the p value by 2 Errors in Hypothesis Testing Process of hypothesis testing is not perfect Never certain of decision Decision is reached on the basis of evidence presented through data, two kinds of errors may occur Errors in Hypothesis Testing H 0 : Innocent Jury Trial Hypothesis Test The Truth The Truth Type I Error p value = probability of making a Type I Error p value < 0.0 Verdict Innocent Guilty Decision H True H 0 0 False Innocent Correct Error Do Not Reject H 0 (no difference) Correct Decision Type II Error Guilty Error Correct Reject H 0 (difference) Type I Error Correct Decision (Power)

14 Large Group Discussion What are the strengths/opportunities of using quantitative methods in evaluation? What are the challenges? What type of inferential analysis might you use for your case studies and why?