CHI-SQUARE TEST. The one-sample t test. Welch s two-sample t test in R. Welch s two-sample t test

Transcription

1 The one-sample t test CHI-SQUARE TEST John Fry Boise State University One common statistic for hypothesis testing is the t statistic t = x µ s2 /N The t test looks at the mean x and variance s 2 of a sample The null hypothesis is that the sample is drawn from a population with mean µ (that is, we expect x µ) If t is high enough, we can reject the null hypothesis and conclude that the sample is not drawn from that population Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 1 Welch s two-sample t test Welch s two-sample t test compares the means of two samples t = x 1 x 2 s 2 1 N 1 + s2 1 N 2 Welch s two-sample t test in R > t.test(c(72,73,76,76,78),c(67,72,76,76,84)) Welch Two Sample t-test data: c(72, 73, 76, 76, 78) and c(67, 72, 76, 76, 84) t = 0, df = 5.202, p-value = 1 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: sample estimates: mean of x mean of y Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 2 Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 3

2 Pearson s χ 2 ( chi square ) test Differences between t and χ 2 tests The most popular hypothesis test in corpus linguistics is the χ 2 ( chi square ) test The χ 2 test compares a set of observed frequencies O with a set of expected frequencies E χ 2 = (O E) 2 E The t-test compares the means of continuous (interval or ratio) variables (e.g., height, weight, rainfall) The χ 2 test is for the observed frequencies of nominal (categorical) variables (e.g., male vs. female) The t test assumes that the population is normally distributed If the difference between observed and expected frequencies is large, we can reject the null hypothesis of independence H 0 : χ 2 = 0 H 1 : χ 2 > 0 Normality is a reasonable assumption in many empirical sciences, but probably not corpus linguistics (cf. Zipf s Law) Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 4 Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 5 χ 2 example: phrasal verbs χ 2 test in R In phrasal verbs, the object (O) and particle (P) can alternate VOP construction brought the book back VPO construction brought back the book Is one construction more common than the other? Null hypothesis: both constructions are equally frequent Say we looked in a large corpus and found the VOP pattern (e.g., brought the book back) is more frequent VPO VOP Observed Are these results statistically significant? Running the χ 2 test in R > chisq.test(c(194, 209)) Chi-squared test for given probabilities data: c(194, 209) X-squared = , df = 1, p-value = Interpretation: The difference is statistically insignificant (χ 2 = 0.56; df = 1; p = 0.455), so we must assume the two constructions are equally frequent in the population for which the sample is representative Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 6 Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 7

3 χ 2 test in R Note that chisq.test only needs the vector of observed frequencies; it computes the expected frequencies itself Use str to see the structure of the test result > str(chisq.test(c(194, 209))) List of 8 $ statistic: Named num attr(*, "names")= chr "X-squared" $ parameter: Named num 1..- attr(*, "names")= chr "df" $ p.value : num $ method : chr "Chi-squared test for given probabilities" $ data.name: chr "c(194, 209)" $ observed : num [1:2] $ expected : num [1:2] $ residuals: num [1:2] attr(*, "class")= chr "htest" χ 2 example: word frequencies in Moby Dick # Read in Moby Dick and tokenize it > moby <- scan(what="c", sep="\n", file="melville-moby_dick.txt") Read items > moby <- tolower(moby) > words <- unlist(strsplit(moby, "\\W+")) > tokens <- words[words!= ""] # Is the observed frequency of each word type in Moby Dick # essentially the same? > chisq.test(table(tokens)) Chi-squared test for given probabilities data: table(tokens) X-squared = , df = 16873, p-value < 2.2e-16 # We reject the null hypothesis that all word types are # equally frequent. Duh! Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 8 Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University contingency tables Example: treatment vs. placebo Another form of the χ 2 test is for a 2 2 contingency table, containing observed frequencies W, X, Y, and Z Condition C Condition C Result R W X Result R Y Z The conditions and results are both categorical variables, such as treatment vs. placebo, or male vs. female For such a 2 2 contingency table, we compute χ 2 as χ 2 = (W + X + Y + Z)(W Z XY )2 (W + X)(W + Y )(X + Z)(Y + Z) Step 1: assemble our data into a 2 2 contingency table Placebo Treatment Number of people cured Number of people not cured Step 2: calculate χ 2 as follows: χ 2 = ( )( )2 ( )( )( )( ) Step 3: determine significance = 2.39 If χ 2 > 3.841, then we can reject the null hypothesis with 95% confidence (p < 0.05) Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 10 Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 11

4 Interpreting χ 2 results χ 2 table for a 2 2 contingency table: α χ Treatment vs. placebo data in R We put the data in the form of a 2x2 matrix > chisq.test(matrix(c(25,60,35,51), nrow=2)) data: matrix(c(25, 60, 35, 51), nrow = 2) X-squared = , df = 1, p-value = Without the correction, results match our hand calculation The p-value is the probability of obtaining a result at least as extreme as a given data point, under the null hypothesis One rejects the null hypothesis only if p is smaller than or equal to a previously chosen significance level α > chisq.test(matrix(c(25,60,35,51), nrow=2), correct=f) Pearson s Chi-squared test data: matrix(c(25, 60, 35, 51), nrow = 2) X-squared = , df = 1, p-value = Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 12 Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 13 Interpreting χ 2 results Example: heaviness of subject NPs χ 2 table for a 2 2 contingency table: α χ In the placebo example, χ 2 = 2.39, which is too small; we cannot reject the null hypothesis The standard significance level used in the social sciences is α = 0.05, but corpus linguists often use the stricter α = 0.01 One quasi-standard way of reporting results: p < 0.05 p < 0.01 p < significant very significant highly significant Aarts (1971) (ch. 4) examined the heaviness of NPs that occur in Subject position in the SEU corpus Light NPs include N, Det N, pronouns, names Heavy NPs contain adjectives, PPs, and other modifiers The null hypothesis is that there is no difference; subject NPs are no lighter or heavier than non-subject NPs Aarts (1971) Table 4.10 (p. 45): Light NP Heavy NP Subject position Non-subject position Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 14 Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 15

5 Example: heaviness of subject NPs Run the χ 2 test in R > chisq.test(matrix(c(6749,1160,4770,4331), nrow=2)) data: matrix(c(6749, 1160, 4770, 4331), nrow = 2) X-squared = , df = 1, p-value < 2.2e-16 The χ 2 value for this table is enormous (p < 0.001), which means we can confidently reject the null hypothesis and conclude that subjects are lighter Phrasal verbs and concreteness In phrasal verbs, the object (O) and particle (P) can alternate VOP construction brought the book back VPO construction brought back the book Gries (2003) looked at whether the object (O) was abstract (like peace) or concrete (like book) Object VPO VOP Abstract Concrete > chisq.test(matrix(c(125,64,69,145), nrow=2)) data: matrix(c(125, 64, 69, 145), nrow = 2) X-squared = , df = 1, p-value = 2.142e-11 Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 16 Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 17 Other uses for the χ 2 statistic Finding translation pairs in aligned corpora cow cow vache 59 6 vache Here χ 2 = , so we conclude these are translation pairs As a metric for corpus similarity (Kilgarriff & Rose 1998) Corpus 1 Corpus 2 word word word Since the count ratios are similar, we cannot reject the null hypothesis that both corpora are drawn from the same source Linguistics 497: Corpus Linguistics, Spring 2011, Boise State University 18