Chapter 7 Part 2. Hypothesis testing Power

Transcription

1 Chapter 7 Part 2 Hypothesis testing Power November 6, 2008 All of the normal curves in this handout are sampling distributions

2 Goal: To understand the process of hypothesis testing and the relationship of sample size and the form of the alternative hypothesis to power. Skills: Will know how and when to conduct an hypothesis test. Will be able to describe the relationship between the hypothesis testing and confidence interval approaches. Will know why power is important and how to maimize it. Contents: Formalization of Hypothesis Testing (Normal distribution and σ Known) 2 2 table Page 1 Two-tailed alternative hypothesis Page 5 Confidence interval approach to two-tailed alternative Page 8 Comparison of processes for hypothesis testing and CI approach Page 9 One-tailed alternative hypothesis Page 9 One-sided confidence interval approach to one-tailed alternative Page 11 Alternative hypothesis a single number rather than a range of numbers Page 12 Figures Figure 1: Two-tailed rejection region Page 6 Figure 2: p-value Page 7 Figure 3: One-tailed rejection region Page 10 Figure 4: Rejection region for single number as the alternative Page 13 Figure 5: Power for a single number as the alternative Page 13

3 Hypothesis Testing - Part 2 Review Riverboat Gambler Hypothesis Testing setup: H 0 is the tested or null hypothesis H A is the alternative hypothesis (sometimes referred to as H 1 ) TRUTH H 0 H 0 is true and H 0 is accepted Correct Decision H A DECISION Accept H A is true and H 0 is accepted Type II or β error Reject H 0 H 0 is true and H 0 is rejected Type I or error H A is true and H 0 is rejected Correct decision - power We also say Fail to reject H 0 " instead of saying Accept H 0 " Accept H 0 " is equivalent to saying Reject H A Reject H 0 " is equivalent to saying Accept H A H 0 is true is equivalent to H A is false H 0 is false is equivalent to H A is true First we are going to review what we learned from the Riverboat Gambler. Hypothesis testing process we used for the Riverboat Gambler: H 0 : p = 0.5 H A : p = 0.2 Notice that we picked as the alternative hypothesis what we believed to be the truth (i.e. if we had believed the coin was fair, we never would have set up the testing of the coin). The null hypothesis is set up as a strawman to be rejected so that we can accept the alternative hypothesis which is our real interest. This is because we have more Page -1-

4 control over the probability of incorrectly accepting the alternative hypothesis. Remember that the probability of incorrectly accepting the alternative hypothesis is equivalent to the probability of rejecting the null hypothesis when in fact it is true (see highlighted cell below). The error associated with this is the Type I or error. But we get to select what we will use as the value of. So although we hope the alternative hypothesis will be true, we certainly don t want to declare it true when it isn t. Type I and Type II errors Let X be the random variable indicating the number of heads in 10 tosses. We selected a critical or rejection region - let us use {0,1,9,10}. (Note that here we have selected the rejection region rather than the level, but as we will see this is not how things are usually done.) Then Pr(X= 0 or 1 or 9 or 10 p = 0.5) = Equation 1 level of significance, level or type I error = (see Table 1 Chapter 7 Part 1) Notice that Equation 1 gives the probability of landing in the rejection region when the null hypothesis (p = 0.5) is true. Pr(X = 0 or 1 or 9 or 10 p= 0.2) = power = 1 - β = Equation 2 Equation 2 gives the probability of landing in the rejection region when the null hypothesis is false (e.g. the alternative hypothesis is true). Page -2-

5 p-value Definition: The p-value is the probability associated with the smallest rejection region that includes the value of the test statistic (the number of heads you actually got in 10 tosses) for the sample, under the assumption that the null hypothesis is true (i.e. the coin tossing problem, assuming the probability of getting heads on a given toss is 0.5). Eample: If turns out to be 0 (i.e. we flipped the coin 10 times and the results of each of the flips was tails), then the smallest rejection region containing 0 is {0, 10} and the p-value is Pr(X = 0 or 10 p = 0.5) = p-value = Notice that the p-value and level have the same pattern (i.e. the probability of being in a rejection region given the null hypothesis is true) just different rejection regions. If the original setup is two-tailed, then the rejection region associated with the p-value is also two tailed (which eplains why we included 10 when calculating the p-value). Eample: If the test statistic turns out to be 4, then the smallest rejection region containing 4 is {0, 1, 2, 3, 4, 6, 7, 8, 9, 10}. That is, the only number not in the rejection region is 5. So p-value = Pr(X = 0, 1, 2, 3, 4, 6, 7, 8, 9, 10 p = 0.5) = 1 - Pr(X = 5 p = 0.5) = = Relationship of the p-value and level Let us go back to the rejection region {0, 1, 9, 10}. For a categorical distribution when the test statistic is in the rejection region [ i.e. = 0 is in the region {0,1,9,10}], the p- value is less than or equal to the level because p-value = Pr(X = 0 or 10 p = 0.5) = for test statistic level = Pr(X= 0 or 1 or 9 or 10 p = 0.5) = = 0 and Page -3-

6 Notice that {, 010} {,,, 01910} Notice that both the p-value and the When the test statistic is not in the rejection region [i.e. region {0,1,9,10}], then the p-value is greater than the statistic = 4 is means is a subset of level are based on the null (p = 0.5) hypothesis. = 4 is not in the rejection level. The p-value for the test p-value = Pr(X = 0, 1, 2, 3, 4, 6, 7, 8, 9, 10 p = 0.5) = as compared to the level = Pr(X= 0 or 1 or 9 or 10 p = 0.5) = So we have that the level and the p-value each depend on the null hypothesis and a rejection region, but not necessarily the same rejection region. The power and β error depend on the alternative hypothesis and the rejection region. We will reject the null hypothesis in favor of the alternative hypothesis if the p-value <. We will fail to reject the null hypothesis, if the p-value >. We fail to reject the null hypothesis when the p-value equals Eample of Hypothesis testing given a normal distribution with σ known (I think that the probability of actually knowing σ for a given study is smaller than my probability of winning the lottery {and I don t buy lottery tickets}, but the assumption simplifies our eample): Let us assume that we know that the population of all high school aged kids in Houston has a mean SBP of 125 mm Hg (Rosner labels this population mean μ 0 ) and a known standard deviation of 50 mm Hg. Suppose we obtain a sample of 25 kids from the High School for the Performing Arts (HSPVA) and find that their mean systolic blood pressure is 142 mm Hg. The question is: Does our sample of 25 kids seem to be from the population with mean SBP = 125 mm Hg or does it seem more likely that they are from another population. So our null hypothesis is: H 0 : μ = 125 mm Hg The μ in the null hypothesis above is the mean for the population from which the sample of 25 kids was drawn. The number 125 mm Hg is the mean of the population of all high school aged kids in Houston (i.e. μ 0 = 125). So we are asking are the means of the two populations the same. If we assume the variances are the same, then we are asking are the two populations the same. Page -4-

7 There are a number of possible forms for the alternative hypothesis: 1) H A : μ 125 (two-tailed alternative) 2) H A : μ > 125 (one-tailed upper tail) 3) H A : μ < 125 (one-tailed lower tail) 4) H A : μ = 162 (some specific value) The form of the alternative hypothesis is picked prior to collecting the sample information. Please note that in the hypothesis testing setup, μ is the mean of the population from which the sample (of 25) was drawn (we hope this population is the same as the one with mean = 125). So μ = 125 essentially asks if the sample was drawn from the same population as the population with mean = 125 (i.e. are the two populations the same). Or another way to think of the question is: is the mean of our sample of 25 consistent with the null hypothesis or with the alternative hypothesis? Let us assume the following: (1) the two-tailed version (i.e. H A : μ 125) of the alternative hypothesis (2) X is the random variable associated with the SBP of all high school kids in Houston (as opposed to the sample from HSPVA). (3) X ~ N( 125, 50 2 ) (4) the level of significance is 0.05 (i.e. = 0.05). Note that to select and then determine the rejection region formed by that level is the usual procedure. In the Riverboat gambler problem we selected the rejection region and then calculated for pedagogical reasons. The sample mean ( = 142 mm Hg) of our sample of 25 kids from HSPVA is our test statistic just like = 2 heads was the test statistic for the coin toss problem. Under the null hypothesis (i.e. assume that the sample of 25 HSPVA students is from the population of all high school students) the distribution of the sample means of all samples of size n = 25 (i.e. the sampling distribution) with σ = 25 (SD of the population of all Houston kids) is the normal distribution with mean = 125 mm Hg and Page -5-

8 50 25 X ~ N 125, SD = 10 mm Hg (i.e ). Or because X ~ N( 125, 50 2 ) and n, our sample size, is equal to In the Riverboat gambler problem, we selected the rejection region and then found the level that went with it. We did it in that fashion so it would be clear that the rejection region contained the values that we thought were unlikely to occur if indeed the coin was fair. Another way to decide on the rejection region is to first select the level (usually something like 0.05 or 0.01) and then find the region (which would be in two pieces for a two-tailed test or one piece for a one-tailed test) whose area is equal to the level. Selecting first is the usual way of doing things. For the current problem we ll choose the level to be Since we are dealing with a two-tailed test, we are looking for the values that cutoff the upper and lower area of the curve. We know that on the N(0,1) curve and 1.96 cut off the lower and upper areas respectively. The question is what points on the N(125,10 2 ) [the distribution for X] curve are the equivalents of and 1.96 (i.e. what points are 1.96 standard deviations below and above 125). 125 = = If, then = (10)(1.96) = and if, then = (10)(-1.96) = This means that the rejection region consists of all such that < or >144.6 (Notice that the endpoints are in the acceptance region.) and the acceptance region is [105.4, 144.6] where the square brackets indicate that the interval includes the end points. Another way to say this is: Pr( X is in the rejection region μ = 125 and σ = 10) = Pr( X < or X > μ = 125 and σ = 10) = 0.05 Note that 142 (the mean SBP of our sample of 25 kids) falls in the acceptance region (see Figure 1 below) so we would say that we accept the null hypothesis (or that we fail to reject the null hypothesis). Page -6-

9 Yet another way of saying this is that we believe our sample of 25 kids could have come from the normal distribution with μ = 125 and σ = 50 (i.e. the distribution of all Houston kids). Fail to reject the null hypothesis is the usual way statisticians report the results, but you ll never see this in a journal article. Notice that what we did was assume the sample was from the population of all high school kids and then look to see if that made sense or to see if the sample really seemed to fit with the population of all high school students. Figure 1.04 Normal Density with Mean = 125 and SD = 10 The vertical lines are at = & = Normal density with Mean = 125 and SD = 10 The vertical lines are at = and = The lines are at = & = Sample Means Each stripped area is half of the rejection region. Each area = The rejection region is the stripped area (sum of the two areas actually) in Figure 1 above. To find the p-value that goes with this decision, we need to find the area to the right of 142 under the N(125,100) curve and double it because we are dealing with a two-tailed alternative. Remember to find the p-value you need to find the smallest rejection region that includes 142. This would be the rejection region where 142 is the cutoff (see Figure 2 below). To get the probability associated with the region under the N(125,100) curve and to the right of 142 we need to translate 142 to the N(0,1) curve by finding how many standard deviations 142 is from 125. Page -7-

10 is 1.7 SD s from 125 since 10 = 1.70 Figure 2 Normal Density with Mean = 125 and SD = 10 The vertical lines are at = & = (for a level) and = 142 (for half the p-value).04 Normal density with Mean = 125 and SD = 10 The vertical lines are at = and = The dashed line is at 142 The stripped area is half of the p-value Sample Means Each area beyond the solid line is half of the rejection region (i.e ) According to the tables or Stata, the probability to the right of 1.7 is Since we are dealing with a two-tailed test, the p-value = = You could get this using STATA:. di 2*(1 - normal(( )/10)) So we fail to reject the null hypothesis because p-value >. Notice we are using the normal distribution with the hypothesized mean as opposed to the sample mean. Page -8-

11 A Confidence interval approach to the same problem nother way we could look at this problem (H 0 : μ = 125 mm Hg versus H A : μ 125 mm Hg assuming X follows a normal distribution, σ known to be 50 and = 0.05) would be through confidence intervals: The 95% CI for 142 (i.e. you get the confidence interval about the sample value) given that n = 25 and σ for X is 50 (the original distribution - the population of all Houston kids) would be ( ( z σ / n), + ( z σ / n)) 1 ( 2) 1 ( 2) (. 196)( 50) (. 196)( 50) 142, = ( , ) = (122.4, 161.6) Notice that 125 (the null hypothesis) is in this 95% confidence interval for 142, so 142 would not be considered different from 125 at = 0.05 level. This is the same conclusion we reached earlier. Using the confidence interval, we know the set of values (population means) that 142 does not differ from, not just that 142 doesn t differ from 125. However, we can t calculate the p-value. So we gain something with the confidence interval approach and lose something. Comparison of processes for hypothesis testing and confidence interval approach: For the confidence interval approach the process is to find the confidence interval for the sample mean (142) and then to check whether the population mean (125) is in that confidence interval (i.e. you start with the sample). Using the hypothesis testing approach, you start with the distribution of X under the null hypothesis (i.e. you get an acceptance region about the population mean 125) and look to see if the sample mean (142) lies in this acceptance region. Page -9-

12 Same problem but with the alternative hypothesis changed to H A : μ > 125 (i.e. one-tailed upper tailed). Let us work through the problem again this time using the #2 form of the alternative hypothesis (i.e H A : μ > 125) but keep the = 0.05 and σ for X at 50. Selecting this particular form of the alternative hypothesis means that we will reject the null hypothesis only when the sample mean is too big. Note that the alternative hypothesis is selected prior to the collection of the sample data. Now, using the hypothesis testing approach, is our sample mean of 142 from a sample of 25 still consistent with the null hypothesis? We will still use the distribution of sample means (i.e. distribution of X) with mean = 125 and SD = 10, but now the critical region is all in the upper tail (i.e. we reject the null hypothesis and accept the alternative only when the sample mean is too big). So again we have the critical and acceptance regions in terms of the relationship to the population mean (125). This means that the entire 0.05 will be in the upper tail because we reject only if 142 is too big. We know that cuts off the upper 5% of the N(0,1) curve. So if we solve the following, we will have the equivalent number for the N(125,10 2 ) curve. Figure 3 Normal Density with Mean = 125 and SD = 10 The vertical line is drawn at = One-tailed test. Normal density with Mean = 125 and SD = 10 The vertical line is at The stripped area is the rejection region. The area = X = Sample bar Means Page -10-

13 125 = If, then = The vertical line on the graph below is at which cuts off 5% of the upper tail. Notice that is smaller than that cut off in the upper tail. And also smaller than 142 which cuts off (i.e. half the p-value from the two-tailed test) in the upper tail. Notice that 142 (the sample mean) is now in the rejection region (i.e. 142 > ) whereas it was in the acceptance region for the two-tailed test To obtain the p-value we again solve 10 di 1 - normal(1.7) = = 1.70 So the p-value = (note that we do not multiply by 2 because this is a one-tailed test). Since 142 is in the rejection region, we epected the p-value to be less than which is equal to For the two-tailed test the test statistic 142 fell in the acceptance region (so we accepted the null hypothesis or failed to reject the null hypothesis) and the p-value was equal to But with the one-tailed test the test statistic falls in the rejection region (so we reject the null hypothesis) and the p-value is (note that is half of ). Therefore, with the same sample size and the same null hypothesis we manage to go from acceptance of the null hypothesis (two-tailed test) to rejection of the null hypothesis (one-tailed test). Note that this is why people want to use a one-tailed test. But be aware that if you have picked the wrong tail (so that the rejection region is on the lower end of the distribution), you could end up with acceptance with a one-tailed test but rejection with a two-tailed. You also need to be aware that it is seldom appropriate to use a one-tailed test. You need to have prior information (i.e. an earlier study done in your lab or published in a journal) that indicates that if the sample mean is different from the population mean it is because the sample mean is bigger than the population mean (version 2 of the alternative hypotheses) or smaller than the population mean (version 3 of the alternative hypotheses). Also particularly note that the alternative hypothesis has to be selected prior to your seeing the data. Investigators like to gamble that they know which tail is appropriate because, as we will see later, you can use a smaller sample size. DON T DO THIS!!!!!! Now consider the confidence interval approach but this time it will be Page -11-

14 a one-sided confidence interval to go with the one-sided test. So we consider a 90% two-sided consider below because if you consider only one side of a 90% confidence interval and you 10% outside the confidence interval with 5% on each side. The confidence interval is again about the sample mean. The 90% confidence interval for 142 is (142 (1.645)(50),142 + (1.645)(50) ) = (125.55, ) The interval (125.55, 4) is the correct one-sided 95% confidence interval. H A :μ > 125 says we reject the null hypothesis in favor of the alternative hypothesis if the sample mean is too big. This can also be translated as the population mean is too small. The confidence interval is about the sample mean 142. So for the population mean to be too small it must be outside the lower bound of the confidence interval. Notice that the tail of this confidence interval is in the same direction as the cutoff for the one-tailed hypothesis test. The population mean 125 is not in the confidence interval so we reject the null hypothesis. Notice that for the hypothesis test we reject the null hypothesis when the sample mean is too big. For the confidence interval we reject the null hypothesis when the population mean is too small. The conclusions with the two-sided hypothesis test and the two-sided confidence interval should be the same provided the level is the same for both. Similarly for the one-sided hypothesis test and one-sided confidence interval. Revisiting the problem with H A as a single number rather than a range of numbers. Now suppose we use the #4 form of the alternative hypothesis (i.e that H A: μ = 162). Now we have two normal distributions for X each with σ = 10 but one with mean = 125 and one with mean = 162. We should note that this sort of problem is not commonly considered. H 0 : μ = 125 versus H A : μ = 162 Page -12-

15 We are going to do a one-tailed upper tailed test here (we reject the null hypothesis only when the sample mean is too big). Why are we doing a one-tailed test here? We want to know if our sample mean of 142 is consistent with 125 or 162. So we reject the null hypothesis when the sample means looks more like 162 than 125. Well numbers smaller than 125 are not going to cause us to reject the null hypothesis in favor of 162. Only if the numbers are too big would we be willing to think that 162 was more appropriate than 125. With this one-tailed test we can actually show the area that goes with the power. Earlier we found that if = 0.05, the cutoff for this upper tailed region is or Pr( X > μ = 125) = 0.05 Figure 4 Rejection Region Normal Distributions with SD = 10 and Means = 125 and Sampling Distribution according to H 0 Sampling Distribution according to H A The line is = The stripped area is the rejection region. Area = Sample Means Power is the probability of being in the rejection region for the null hypothesis when the alternative hypothesis is true (i.e. correctly rejecting the null hypothesis). So the power for this problem is the probability of being to the right of (i.e. being in the rejection region for the null hypothesis) but under the alternative hypothesis curve (i.e. the one with mean = 162). Page -13-

16 So Pr( X > μ = 162) = = power [see Figure 5 below]. Recall that on the normal curve with mean 162 and SD = 10 is equivalent to ( )/10 = (i.e is 2.1 standard deviations below the mean of 162) Using Stata, we get power = 1 - normal(-2.055) normal(-2.055) gives the are under the curve with μ = 162 and to the left of the line at (i.e. β = 1 - power). It is the area under the rest of the curve (i.e. to the right of ) that is the power (i.e. 1 - β). Figure 5 Power Normal Distributions with SD = 10 and Means = 125 and Sampling Distribution according to H 0 Sampling Distribution according to H A The hatched area is the power = The line is = Sample Means Page -14-

17 You need to be aware that: 1) When we were looking at the problems above as hypothesis testing problems, we used the population parameters (here population mean = 125 and population SE = 10) to obtain a rejection region and then asked was the sample statistic (here = 142) in the rejection region. 2) However, when considering the problems using the confidence interval approach, we obtained the confidence interval about the sample statistic ( = 142) and then asked if the population parameter (i.e. the population mean 125) was in the confidence interval. Why do people select a one-tailed versus a two-tailed test? We saw above that it is possible to reject the null hypothesis using a one-tailed test but fail to reject the null hypothesis using a two-tailed test. So if you were trying to prove that your new drug is better than the standard of care you might be tempted to use a one-tailed test. What are the drawbacks to a one-tailed test? Well you might have guessed the wrong tail. If before we had obtained a sample (and this is the way you are supposed to play the game), you said if your sample of kids differed from the population with respect to SBP, it would be because the SBP of your sample of kids would be too small. This means that in Figure 3 above the rejection region would be the area to the left of = = would be in the acceptance region. The wrong guess has cost you a significant answer.. So Page -15-