Reasoning with Uncertainty More about Hypothesis Testing. P-values, types of errors, power of a test

Transcription

1 Reasoning with Uncertainty More about Hypothesis Testing P-values, types of errors, power of a test

2 P-Values and Decisions Your conclusion about any null hypothesis should be accompanied by the P-value of the test. If possible, it should also include a confidence interval for the parameter of interest. Don t just declare the null hypothesis rejected or not rejected. Report the P-value to show the strength of the evidence against the hypothesis. This will let each reader decide whether or not to reject the null hypothesis.

3 What to Do with a High P-Value A big P-value doesn t prove that the null hypothesis is true, but it certainly offers little evidence that it is not true. Thus, when we see a large P-value, all we can say is that we fail to reject the null hypothesis.

4 P-value and Statistical Significance The P-value gives the reader far more information than just stating that you reject or fail to reject the null. In fact, by providing a P-value to the reader, you allow that person to make his or her own decisions about the test. What you consider to be statistically significant might not be the same as what someone else considers statistically significant. There is more than one alpha level of significance (0.05, 0.01,..) that can be used, but each test will give only one P-value.

5 Significant vs. Important What do we mean when we say that a test is statistically significant? All we mean is that the test statistic had a P-value lower than our alpha level. Don t be lulled into thinking that statistical significance carries with it any sense of practical importance or impact.

6 Low-Fat Diets Flub a Test NY Times, Feb For decades or more, medical lore has suggested that a low-fat diet can yield substantial health benefits. Millions of Americans have tried to reduce the fat in their diets, and the food industry has obligingly served up low-fat products. Yet now comes strong evidence that the war against all fats was mostly in vain. The baffling results came from a $415 million study of almost 49,000 women age 50 to 79 who were tracked for eight years, with repeated exhortations to the low-fat dieters to stick to the regimen.

7 Low-Fat Dietary Pattern and Risk of Invasive Breast Cancer The Women s Health Initiative Randomized Controlled Dietary Modification Trial Intervention: Women (postmenopausal women, aged 50 to 79 years) were randomly assigned to the dietary modification intervention group (40% [n = ]) or the comparison group (60% [n = ]). The intervention was designed to promote dietary change with the goals of reducing intake of total fat to 20% of energy and increasing consumption of vegetables and fruit to at least 5 servings daily and grains to at least 6 servings daily. Comparison group participants were not asked to make dietary changes.

8 Making Errors When we perform a hypothesis test, we can make mistakes in two ways: Type I error Rejecting the null hypothesis when it is true. Type II error Failing to reject the null hypothesis when it is false

9 Making Errors Here s an illustration of the four situations in a hypothesis test:

10 Courtroom Analogy: Potential choices and errors Null hypothesis: Defendant is innocent. Alternative hypothesis: Defendant is guilty. Choice A: We cannot rule out that defendant is innocent, so he or she is set free without penalty. Potential error: A criminal has been erroneously freed. Choice B: We believe enough evidence to conclude the defendant is guilty. Potential error: An innocent person falsely convicted and guilty party remains free. Which type of error is more serious?

11 Example: A statistical test in terms of a diagnostic test for a disease Null hypothesis: You do not have the disease Alternative: You have the disease The table shows that when you run a statistical test, you can only make one of two errors: Reject the null hypothesis when it is true (Type I error or a false positive) Fail to reject the null hypothesis when it is false (Type II error a false negative)

12 Probabilities Associated with Errors How often will a Type I error occur? Since a Type I error is rejecting a true null hypothesis, the probability of a Type I error is our α(alpha) level of significance. Example: α = 0.05, we are willing to accept an 1 in 20 chance of making the mistake of rejecting the null in favor of the alternative when the null is true. Our p-value is our observed level of significance. Example: p-value = There is only a 1 in 1000 chance that we made the mistake of rejecting the null when it is true.

13 Probabilities Associated with Errors When H 0 is false and we fail to reject it, we have made a Type II error. We assign the letter β(beta) to the probability of this mistake. It s harder to assess the value of β because we don t know what the value of the parameter really is. There is no single value for β--we can think of a whole collection of β s, one for each incorrect parameter value.

14 Probabilities Associated with Errors When H 0 is false and we reject it, we have done the right thing. A test s ability to detect a false null hypothesis (or a true alternative hypothesis) is called the power of the test. In other words, The power of a test is the probability of making the correct decision when the alternative hypothesis is true.

15 The Power of a Test The power of a test is the probability of making the correct decision when the alternative hypothesis is true. The power of a test is 1 β ; because β is the probability that a test fails to reject a false null hypothesis (Type II error) and power is the probability that it does reject.

16 The Power of a Test and Type II Errors In the Election Example 1, if only 51% of all voters would vote for Candidate A, then a sample of 50 (or even 500) voters could result in a sample proportion that would lead to not rejecting the null hypothesis (true proportion = 0.50 or 50%) when it is in fact false (very high probability of making a type II error test has low power).

17 The Power of a Test and Type II Errors However, if 80% of all of all voters would vote for Candidate A, the sample proportion will almost surely be large enough to reject the null hypothesis in favor of the alternative (very low probability of making a type II error - test has high power). Problem: Impossible to know what part of the alternative (51%, 56%, 65%, 80% and so on) is true.

18 The Power of a Test Medical Testing Example The power of a test is the probability of detecting that a person has the disease when they truly have it, a true positive Election Example 1 The ability to detect that a majority of all voters would vote for Candidate A will be much higher if that majority constitutes 80% than if it constitutes just 51% of the population. If the population value falls close to the value specified in null hypothesis, then it is difficult to get enough evidence from the sample to conclusively choose the alternative hypothesis. There will be a relatively high probability of making a type II error, and the test will have relatively low power in that case.

19 The Power of a Test Whenever a study fails to reject its null hypothesis, the test s power comes into question. When we calculate power, we imagine that the null hypothesis is false. The value of the power depends on how far the truth lies from the null hypothesis value. Power depends directly on the distance between the null hypothesis value, p 0, and the truth, p.

20 The Power of a Test This diagram shows the relationship between these concepts:

21 Reducing Both Type I and Type II Error The previous figure seems to show that if we reduce Type I error, we must automatically increase Type II error. But, we can reduce both types of error by making both curves narrower. How do we make the curves narrower?

22 Reducing Both Type I and Type II Error This figure has means that are just as far apart as in the previous figure, but the sample sizes are larger, the standard errors are smaller, and the error rates are reduced:

23 Reducing Both Type I and Type II Error Original comparison of errors: Comparison of errors with a larger sample size:

24 The Power of a Test - Calculating Sample Size Medical Testing Example: Detecting that a person has the disease when they truly have it, a true positive. The typical question that scientists ask isn't, "OK, I've got 100 people here. What are my chances of a positive result?" but, "I want to test a hypothesis. How many people will I need?" This sort of question is particularly important in medical research. In a trial of a new drug, for example, you want to have a good chance of a statistically significant result if the drug is effective because it would be great to have another way to help sick people. But you can't have too large a sample size - drugs often have side effects, so you don't want to give a new drug to lots of people if it doesn't work.

25 The Power of a Test - Calculating Sample Size A typical question asked of statisticians in medical research might be something like: About 20% of patients recover from a typical cold within 36 hours. We think that our new cold treatment might increase this to 30%. How big a sample do we need to avoid making a type II error failing to reject the null hypothesis when it is false detecting a true difference. As a general rule, we use the usual alpha of 5% and a power of 80%, that is, if the drug is effective, we want a 80% chance of showing that it does indeed work. How many patients do we need?

26 SAS Computing Sample Size title "Computing Sample Size for a Difference in Two Proportions"; proc power; twosamplefreq test = pchi groupproportions = power =.80 npergroup =.; run;

27 SAS Computing Sample Size

28 Question