AMS 5 HYPOTHESIS TESTING

Transcription

1 AMS 5 HYPOTHESIS TESTING

2 Hypothesis Testing Was it due to chance, or something else? Decide between two hypotheses that are mutually exclusive on the basis of evidence from observations.

3 Test of Significance A certain brand of tobacco has an average nicotine content of 2.5 milligrams, while another one has an average of only 1.5 milligrams. A cigarette manufacturer receives an unlabeled shipment of tobacco and needs to determine the nicotine content using a sample of the tobacco. A certain type of vaccine is known to be only 25% effective over a period of 2 years. A new type of vaccine is being tested on 2000 people chosen at random. We want to test if this new vaccine is more effective than the original one.

4 Test of Significance A machine for filling bottles of soda has to put 333 ml of liquid in each bottle. If the average amount is too low or too high with respect to the expected content then the machine is considered to be out of control. The machine is regularly inspected to check whether it is out of control by taking a sample of bottles. A balance designed for precision weighting in a lab has to be kept very well calibrated. To test the calibration of the balance several measurements of the weight of an object are done. If they differ too much then the balance has to be re-calibrated. In all these examples a decision has to be taken based on the numbers in a sample. These numbers are subject to uncertainty and we have to decide if the differences that we observe are only due to chance or not.

5 Null and Alternative Hypotheses Null and Alternative Hypotheses A bill is proposed to simplify the tax code. The proposer claims that the bill is `revenue-neutral', that is, it will not lower tax revenues. A simulation is run using 100 tax returns chosen at random and the differences between the tax paid using the old rules and those that would be paid using the new rules are recorded. The average difference comes up to be -$219 with a standard deviation of $725. Can we claim that the new rules are revenue-neutral? We can put the problem in these terms: there are two hypotheses: NULL HYPOTHESIS, H 0 and ALTERNATIVE HYPOTHESIS, H 1 Under the null hypothesis there is no difference in revenue and the fact that the observed value is not 0 is totally due to chance. Under the alternative hypothesis the difference is real.

6 Null and Alternative Hypotheses For the examples that we considered at the beginning we have that: Cigarette: H 0 : the mean nicotine content is 1.5. H 1 : the mean nicotine content is 2.5 Vaccine: H 0 : the proportion is 25%. H 1 : the proportion is higher than 25%. Soda Bottles: H 0 : the average amount of liquid is 333 ml. H 1 : the average amount of liquid is not equal to 333 ml. Balance: H 0 : the device is calibrated. H 1 : the device is not properly calibrated.

7 Test Statistics How do we test the null hypothesis against the alternative? Back to the tax example. Suppose the null is true. Then the difference should be $0. How `large' is -$219 with respect to $0? To answer this question we convert to standard units. Given that the sample was of size 100 the SE is approximately $72: $219 $0 $72 3 so the difference between the value under the null and the observed value is -3 standard units. The probability of the interval to the left of -3 is about 0.001, that is, one chance in 1,000. So, under the null hypothesis, $219 is a very unlikely value.

8 Test Statistics In general we are calculating a test statistics given by z = observed exp ected SE which is referred to as the z-test. Once the z-test is calculated we have to decide whether its value is `large' or is `small'. We observe the probability of the left tail of the normal curve, below the z-test. If this probability is small then the value of the z-test is far from the center of the distribution. This probability is called an observed significance level

9 Test Statistics The smaller the P-value, the stronger the evidence against the null, but

10 Making a test of significance To make a test of significance you need to: set up the null hypothesis pick a test statistics to measure the difference between the data and what is expected under the null hypothesis compute the test statistics and the corresponding observed significance level. A small observed significance level implies that the data are far from the values expected from the model under the null hypothesis. What is a small observed significance level? This is somewhat arbitrary, but it is usually considered that if P is less than 5% the results are significant. If P is less than 1% the results are highly significant.

11 Examples 1. A random sample of 85 8 th graders has a mean score of 265 with an SD of 55 on a national math test. A State Administrator claims that the mean score of 8th graders on the examinations is above 260. Is there enough evidence to support the administrator's claim? The hypotheses are: H 0 : mean 260 vs H 1 : mean > 260 The test statistics is obtained by changing to standard units: / 85 = The probability that a standard normal is above is about 21%. This is a rather large P-value, so there does not seem to be enough evidence to reject H 0.

12 Examples 2. A light bulb manufacturer guarantees that the mean life of the bulbs is at least 750 hours. A random sample of 36 light bulbs has a mean of 725 hours and a standard deviation of 60 hours. Is there enough evidence to reject the manufacturer's claim? The hypotheses are H 0 : mean 750 vs H 1 : mean < 750 The test statistics is obtained by changing to standard units: / 36 = 2.05 The probability that a standard normal is below is about 2%. There is some evidence to reject the manufacturer's claim.

13 Binary boxes Consider again the problem of testing the new vaccine. This is a binary model since we can classify the population in two groups: the group of people for which the vaccine was effective and that for which the vaccine was not. Under the null hypothesis the box model that generates the sample consists of the box since there is 25% chance that the vaccine is effective. Suppose that the number of people in the sample (of 2000 people) for which the vaccine was effective is 534. According to the null hypothesis the expected number would be 500. Is the 34 people difference large enough to reject the null hypothesis and claim that the new vaccine is more effective?

14 Binary boxes We need to calculate the standard units of the difference between 550 and 534. Under the null hypothesis the SD of the box is SD= so the standard error is SE= = Then: z= = The observed significance level is given by the area under normal curve corresponding to interval above This is around 4%, which is small enough to conclude that the difference is statistically significant. So there is evidence to support the claim that the new vaccine is more effective than the standard.

15 The t-testt test The examples that we have seen so far rely on the fact that the sample size is large. So, even when the SD of the box is unknown, we can still use the normal curve to obtain the observed significance level of the test. This is not the case when the sample size is small. In this case we need a modification of the z-test due to `Student', a pen name for a statistician called Gosset. Consider the following example. The following five measures of the concentration of Carbon monoxide (CO) are taken from a gas sample where the concentration is precisely controlled to be 70 parts per million (ppm). Five measurements are taken to check the calibration of an instrument

16 The t-testt test The null hypothesis is that the device is calibrated and so the average of the measurements is 70 ppm. The average of the sample is 77.8 ppm, the SD is 7.22 ppm and thus the SE of the average is 7.22 / The z-test can be obtained as z= To determine the observed significance level we calculate the area to the right of 2.4 under the normal curve. This is less than 1%, which looks like strong evidence against the null hypothesis. Unfortunately we have to remember that the SD that we have calculated is NOT the SD of the box. It is the SD of the sample, whose size is fairly small, and thus the approximation is not very precise. We correct the procedure with the following steps.

17 The t-testt test Step 1: Consider a different estimate of the SD number of measurements SD + = SD number of measurements 1 Notice that SD + > SD. 5 In our previous example we get SD + = so the SE of the average becomes 8.07 / 5= 3.61, 4 as opposed to 3.23, reflecting a higher level of uncertainty. Then the test statistics becomes t=

18 The t-testt test Step 2: To find the observed significance level we can not use the normal curve any more. We need to use a Student's t curve. This curve depends on the degrees of freedom (DF). These are calculated as degrees of freedom = number of measurements - 1 A table for the Student's t curves is found at the end of the book. There is one curve for each value of the DF. Each row corresponds to one curve. The probabilities that are reported correspond to the right hand tail, as opposed to what was reported for the normal curve. These curves are symmetric around 0 and for DF above 25 they resemble the normal curve very closely. Thus, in our example, we need a Student's t curve with 4 DF. The value 2.2 is not present in the table for the row corresponding to 4 DF. The closest value is 2.13, which corresponds to 5%. So the P-value for this test is about 5%. Which is much weaker an evidence against the null than before.

19 The t-testt test Suppose now that 6 measures are taken with the device The average is equal to 74 ppm and the SD is 6.68 ppm. The corrected SD is SD + = 6 / The SE of the average is So the t-test is 7.23/ t= This time the DF are 5 and if we look at the table we find that the probability corresponding to 1.48 is 10%. Since 1.34 is smaller than 1.48 we have that the P-value is larger than 10%. This is not enough evidence against the null. So the machine can be considered to be well calibrated.

20 Examples 1. An environmentalist estimates that the mean waste recycled by adults in the US is more than 1 pound per person per day. You take a sample of 12 adults and find that the waste generated per person per day is 1.2 pounds with a standard deviation of 0.3 pounds. Can you support the environmentalist's claim? The hypotheses are: H 0 : mean 1 vs H 1 : mean > 1 The corrected value of the SD is 12 /11 0.3= 0.32 and the test statistics is obtained by changing to standard units: / 12 = 2.17 The probability that a Student with 11 degrees of freedom will be above 2.17 is about 2.5%. This is a rather small P-value, so there seems to be enough evidence to reject H 0.

21 Examples 2. A microwave oven repairer says that the mean repair cost for damaged microwave ovens is less than $100. You find a random sample of 5 ovens has a mean repair cost of $75 with an SD of $12.5. Do you have enough evidence to support the repairer's claim? H 0 : mean 100 vs H 1 : mean < 100 The corrected value of the SD is 5/ = and the test statistics is obtained by changing to standard units: / 5 = 4.01 The probability that a student with 4 degrees of freedom will be above 4.01 is less than 1%. This is a rather small P-value, so there seems to be enough evidence to reject H 0.

22 Was the result significant? How small does P-value P have to get before you reject the null hypothesis? If P-value P < 5% statistically significant. If P-value P < 1% highly significant. Therefore a P-value P of 5.1% is totally different than a P-value P of 4.9%!!!!! Investigators should summarize the data, say what test was used, and report the P-value P instead of just comparing it to 5% or 1%

23 Data Snooping A result which is statistically significant cannot be explain as chance variation. This is false. Even if the null hypothesis is right, there is a 5% chance of getting a difference which the test will call statistically significant! Therefore an investigator who makes 100 tests expect to get 5 results which are statistically significant due to chance!!! To make bad enough worse, investigators often decide which hypotheses to test only after have seen the data data snooping.

24 Practical significance Statistical significance and practical significance are two different ideas!!! The P-value P of a test depends on the sample size. With a large sample, even a small difference can be statistically significant, that is, hard to explain by the luck of the draw. This doesn t necessarily make it important. Conversely, an important difference may not be statistically significant if the sample is too small.