ORF 245 Fundamentals of Statistics Chapter 8 Hypothesis Testing

Transcription

1 ORF 245 Fundamentals of Statistics Chapter 8 Hypothesis Testing Robert Vanderbei Fall 2015 Slides last edited on December 11, rvdb

2 Coin Tossing Example Consider two coins. Coin 0 is fair (p =0.5) butcoin1isbiasedinfavorofheads:p =0.7. Imagine tossing one of these coins n times. The number of heads is a binomial random variable with the corresponding p values. The probability mass function is therefore p(x) = n! p x (1 p) n x x For n =8,theprobabilitymassfunctionsare x p(x H 0 ) p(x H 1 ) Suppose that we know that our coin is either the fair coin or the biased coin. We flip it n =8times and it comes up heads 3 times. The probability of getting 3 heads with the fair coin is p(3 H 0 )= whereas the probability of getting 3 heads with the biased coin is p(3 H 1 )= Hence,itis0.2188/ = 4.69 times more likely that our coin is the fair coin than it is the biased coin. The ratio p(x H 0 )/p(x H 1 ) is called the likelihood ratio: x p(x H 0 ) p(x H 1 ) 1

3 Bayesian Approach We consider two hypotheses: H 0 = Coin 0 is being tossed and H 1 = Coin 1 is being tossed Hypothesis H 0 is called the null hypothesis whereas H 1 is called the alternative hypothesis. Suppose we have a prior believe as to the probability P (H 0 ) that coin 0 is being tossed. Let P (H 1 )=1 P (H 0 ) denote our prior belief regarding the probability that coin 1 is being tossed. A natural initial choice is P (H 0 )=P(H 1 )=1/2. If we now toss the coin n times and observe x heads, then the posterior probability estimate that coin 0 is being tossed is P (H 0 x) = p(x H 0 )P (H 0 ) p(x H 0 )P (H 0 )+p(x H 1 )P (H 1 ) Hence, we get a simple intuitive formula for the ratio P (H 0 x) P (H 1 x) = P (H 0) P (H 1 ) p(x H 0 ) p(x H 1 ) 2

4 Making a Decision If we must make a choice, we d probably go with H 0 if P (H 0 x) P (H 1 x) = P (H 0) P (H 1 ) p(x H 0 ) p(x H 1 ) > 1 and H 1 otherwise. This inequality can be expressed in terms of the likelihood ratio: where c denotes the ratio of our priors. p(x H 0 ) p(x H 1 ) > P (H 1) P (H 0 ) =: c If we use an unbiased prior, then c =1and we will accept H 0 if X apple 4 and we will reject H 0 is X>4. 3

5 Type I and Type II Errors With c =1,wewillacceptH 0 if X apple 4 and we will reject H 0 if X>4. Type I Error: Reject H 0 when it is correct: P (reject H 0 H 0 ) = P (X >4 H 0 ) = Type II Error: Accept H 0 when it is incorrect: P (accept H 0 H 1 ) = P (X apple 4 H 1 ) = With c =5,wewillacceptH 0 if X apple 2 and we will reject H 0 is X>2. Inthiscase,weget Type I Error: P (reject H 0 H 0 ) = P (X >2 H 0 ) = Type II Error: P (accept H 0 H 1 ) = P (X apple 2 H 1 ) = The parameter c controls the trade-o between Type-I and Type-II errors. 4

6 Type-I and Type-II Errors PP P PPPPPP Reality Decision H 0 true H 1 true Accept H 0 Yes! Type-II error Reject H 0 Type-I error Yes! AType-Ierrorisalsocalledafalse discovery or a false positive. AType-IIerrorisalsocalledamissed discovery or a false negative. 5

7 Neyman-Pearson Paradigm Abandon the Bayesian approach. Instead, focus on probability of Type-I and Type-II errors associated with a threshold c. Things to Consider It is conventional to choose the simpler of two hypotheses as the null. The consequences of incorrectly rejecting one hypothesis may be graver than those of incorrectly rejecting the other. In such a case, the former should be chosen as the null hypothesis, because the probability of falsely rejecting it could be controlled by choosing. Examples of this kind arise in screening new drugs; frequently, it must be documented rather conclusively that a new drug has a positive e ect before it is accepted for general use. In scientific investigations, the null hypothesis is often a simple explanation that must be discredited in order to demonstrate the presence of some physical phenomenon or e ect. In criminal matters: Innocent until proven guilty! 6

8 Terminology Null Hypothesis: H 0 Alternative Hypothesis: H 1 Type I Error: Rejecting H 0 when it is true. Significance Level: The probability of a Type-I error. Usually denoted. Type II Error: Accepting H 0 when it is false. Probability usually denoted. Power: 1. Test Statistic: Rejection region: Acceptance region: The measured quantity on which a decision is based. The set of values of the test statistic that lead to rejection of the null hypothesis. The set of values of the test statistic that lead to acceptance of the null hypothesis. Null Distribution: p(x H 0 ) 7

9 Unrealistic Example Let X 1,X 2,...,X n be a random sample from a normal distribution having known variance 2 but unknown mean µ. Weconsidertwohypotheses: H 0 : µ = µ 0 H 1 : µ = µ 1 where µ 0 and µ 1 are two specific possible values (suppose that µ 0 <µ 1 ). The likelihood ratio is the ratio of the joint density functions: 0 nx (x i 1 µ 0 ) 2 /2 2 A f 0 (x 1,...,x n ) f 1 (x 1,...,x n ) = i=1 0 1 nx (x i µ 1 ) 2 /2 2 A We will accept the null hypothesis if this ratio is larger than some positive threshold. Let s call the threshold e c and take logs of both sides: i=1 nx (x i µ 0 ) 2 /2 2 + i=1 nx (x i µ 1 ) 2 /2 2 >c i=1 8

10 Example Continued Expanding the squares, combining the two sums, and simplifying, we get nx i=1 Dividing both sides by 2n and by µ 1 2x i (µ 0 µ 1 ) µ µ 2 1 µ 0,weget > 2 2 c x +(µ 0 + µ 1 )/2 > 2 c/n(µ 1 µ 0 ) where x = 1 n P n i=1 x i.finally,isolating x from the other terms, we get x <(µ 0 + µ 1 )/2 2 c/n(µ 1 µ 0 ) Choosing c =0corresponds to equally probable priors (since e 0 =1). In this case, we get the intuitive result that we should accept the null hypothesis if x <(µ 0 + µ 1 )/2 From these calculations, we see that the test statistic is the sample mean X = 1 n P n i=1 X i and that we should reject the null hypothesis when the test statistic is larger than some threshold value, let s call it x 0. 9

11 Example Significance Level In hypothesis testing, one designs the test according to a set value for the significance level. In this problem, the significance level is given by = P ( X >x 0 H 0 ) If we subtract µ 0 from both sides and then divide by astandardnormaldistribution: 0 1 = X µ 0 / p n > x 0 µ 0 / p n H 0 / p n,wegetarandomvariablewith x A 0 µ 0 =1 F N / p n where F N denotes the cumulative distribution function for a standard Normal random variable. In a similar manner, we can compute the probability of a Type-II error:! = P ( X apple x 0 H 1 )=P X µ 1 / p n apple x 0 µ 1 / p n H 1 1! x A 0 µ 1 = F N / p n 10

12 Example vs. Trade-O µ 0 = 2 µ 1 = π β α x 0 As x 0 slides from left to right, goes down whereas goes up. The sum + is minimized at x 0 =(µ 0 + µ 1 )/2. To make both and smaller, need to make / p n smaller; i.e., increase n. 11

13 Generalized Likelihood Ratios Let X 1,...,X n be iid Normal rv s with unknown mean µ and known variance 2. We consider two hypotheses: H 0 : µ = µ 0 H 1 : µ 6= µ 0 The null hypothesis H 0 is simple. ThealternativeH 1 is composite. For simple hypotheses, the likelihood that the hypothesis is true given specific observed values x 1,x 2,...,x n is just the joint pdf. Hence, the likelihood that H 0 is true is given by: f 0 (x 1,...,x n )= 1 ( p 2 ) ne P n i=1 (x i µ 0 ) 2 For composite hypotheses, the generalized likelihood is taken to be the largest possible likelihood that could be obtained over all choices of the distribution. So, the likelihood that H 1 is true is given by: f 1 (x 1,...,x n ) = max 1 ( p 1 P n 2 ) ne 2 2 i=1 (x i ) 2 MLE = 1 ( p 2 ) ne P n i=1 (x i x) 2 NOTE: The maximum is achieved by the maximum likelihood estimator, whichwehave already seen is just x. 12

14 Likelihood Ratios Continued As before, we define our acceptance region based on the log-likelihoood-ratio: (x 1,...,x n ) = log f 0(x 1,...,x n ) f 1 (x 1,...,x n ) = nx i=1 (x i µ 0 ) nx (x i x) 2 Note that f 0 apple f 1 and therefore apple 0. We accept the null hypothesis when is not too negative: (x 1,...,x n ) > for some negative constant: c. Again as before, we expand the quadratics and simplify the inequality to get nx i=1 2x i (µ 0 x) µ x 2 > 2 2 c Now, we use the fact that P i x i = n x to help us further simplify: 2 x(µ 0 x) µ x 2 > 2 Some final algebraic manipulations and we get: 2 c 2 n c ( x µ 0 ) 2 < 2 n c 13 i=1

15 Likelihood Ratios Continued And, switching to capital-letter random-variable notation, we get that we should accept H 0 if X µ 0 < p 2cp n or, in other words, µ 0 p 2cpn < X < µ 0 + p 2cp n 14

16 More Realistic Example As before, let X 1,X 2,...,X n be a random sample from a normal distribution having known variance 2 but unknown mean µ. This time, let s consider these two hypotheses: where µ 0 is a specific/given value. Reject null hypothesis if H 0 : µ = µ 0 H 1 : µ 6= µ 0 X µ 0 >x 0 where x 0 is chosen so that P H0 X µ 0 >x 0 = (note the probability is computed assuming H 0 is true). Since we know that the distribution is normal, we see that x 0 = z( /2) X 15

17 Confidence Interval vs Hypothesis Test It s easy to check that P H0 X µ 0 >z( /2) X = is equivalent to P H0 X z( /2) X apple µ 0 apple X + z( /2) X =1 The 100(1 )% confidence interval for µ is P µ X z( /2) X apple µ apple X + z( /2) X =1 Comparing the acceptance region for the test to the confidence interval, we see that µ 0 lies in the confidence interval for µ if and only if the hypothesis test accepts the null hypothesis. In other words, the confidence interval consists precisely of all those values of µ 0 for which the null hypothesis H 0 : µ = µ 0 is accepted. 16

18 p-values 17

19 Local Climate Data Forty Year Di erences On a per century basis... X =4.25 F/century, S =26.5 F/century, S/ p n =0.35 F/century Hypotheses: Number of standard deviations out: temperature Forty Year Temperature Differences date H 0 : µ =0, H 1 : µ 6= 0 z = X/S X =4.25/0.35 = 12.1 Probability that such an outlier happened by chance: p = P (N >12.1) = Now I m way convinced! Caveat: The random variables are not fully independent. If it s warmer than average today, then it is likely to be warmer than average tomorrow. Had we used daily averages from just one day a week, the variables would be closer to independent. We d still reject the null hypothesis but z would be smaller (by a factor of p 7)andsothep value would be larger. 18